** Description changed: When testing natty I've noticed many times when a test such as the wifi slider locks up the system, checkbox will not resume the test where it was left off. It will restart at the beginning of the test run after rebooting the system, causing a lot of wasted time testing. + + [Impact] + Since job information may be kept in memory and/or disk cache, it's possible for checkbox to start executing a test which crashes the system before the data is written to disk. When this happens, it often means that all test data up to the point of failure is lost, and the tester is forced to start the run from scratch, wasting time. + + The fix for this bug improves reliability by requesting to flush all + relevant caches prior to test execution. As mentioned in comment #10, + this greatly reduces the chance that test data will be lost. + + Since we depend on the goodwill of users to run Ubuntu Friendly tests, + at least ensuring that their work doesn't go to waste if the system + crashes is a good rationale for requesting rolling this fix into the + stable release. + + [Development fix] + Checkbox revision 1090 implements a fix by adding a safe_close method that takes care of flushing file descriptors and syncing the disk cache, and gets called before starting a job run. + + [Stable fix] + This is fixed in http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/revision/1090 which applies cleanly to the stable release of checkbox (0.12.8). Note that a changelog entry for this bug was not generated. I'm attaching a patch which includes the changelog entry for completeness, but otherwise is identical to the changes in rev 1090. + + [Test Case] + - Start "system testing" (checkbox-gtk version 0.12.8) on an 11.10 system using Unity 3d. + - Press "Next" and provide password when prompted. + - Leave all tests and suites selected and press Next. + - When the tests start, just press "alt-X" to invoke the "Next" button, skipping all the manual tests. + - Wait for a long-running test to start. cpu/clocktest is a good candidate. Monitoring .cache/checkbox/checkbox.log for the test names that are running is one way to do it, because bug 868995 may cause checkbox to hide, which makes it difficult to tell what's happening. + - As soon as the test starts, yank power to the system. + - Reboot the system, and try to run system testing again. + + Expected behavior: + - Checkbox should offer to continue the interrupted run + - If accepted, it should go back to the last test that was running (e.g. cpu/clocktest) and continue from there. + Actual behavior: + - Checkbox may not offer to continue an interrupted run, showing that it has no record of one having been started. + - Even if offering to continue the run, checkbox may not start where it was interrupted, losing some test results that have to be repeated. + + [Regression potential] + The code is rather simple, but it could cause some perceived slowness during test execution, as checkbox will now insist on flushing everything to disk before each test. Also it's possible that this extra disk activity may trigger latent disk failures that can potentially be blamed on checkbox, but the level of extra activity is comparatively small and should pose no actual problems.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/814801 Title: Checkbox loses results on system freeze To manage notifications about this bug go to: https://bugs.launchpad.net/checkbox/+bug/814801/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
