Great stuff! Good work Nicolas. Erik
On Tue, Oct 6, 2009 at 5:02 PM, Nicolas Sylvain <[email protected]> wrote: > Hello, > We currently have more than 50 unit tests that are disabled. Most of them > because they were flaky. > Disabling tests is bad because we lose complete coverage on them, so I > implemented a way to mark > tests as "flaky". > The same way you disable a test with DISABLED_ at the beginning of its name, > you can now mark > is as flaky with FLAKY_. The behavior is exactly the same as any other > running tests. You will still > be able to see when it fails (and why). The only difference is that if only > FLAKY_ tests failed, the > buildbot/trybots won't consider it as a failure. On the waterfall, it will > show the box as orange with the > list of all flaky tests that failed (pending one more buildbot restart). On > the console view it will stay > green. > But.. this is not a toy. Flaky tests are bad. We should mark tests flaky > only if we really have to, and > if you do, please make sure to file a P1 bug. Set the owner of the bug to > whoever regressed the test. > If you can't find who regressed the test, assign it to the person who > originally wrote the test. > Once we start tagging the flaky tests, we will monitor the flakiness > dashboard and make sure > that a test that is no longer flaky has its FLAKY_ tag removed. > Let me know if you have questions. > Thanks > Nicolas > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
