Hello, We currently have more than 50 unit tests that are disabled. Most of them because they were flaky.
Disabling tests is bad because we lose complete coverage on them, so I implemented a way to mark tests as "flaky". The same way you disable a test with DISABLED_ at the beginning of its name, you can now mark is as flaky with FLAKY_. The behavior is exactly the same as any other running tests. You will still be able to see when it fails (and why). The only difference is that if only FLAKY_ tests failed, the buildbot/trybots won't consider it as a failure. On the waterfall, it will show the box as orange with the list of all flaky tests that failed (pending one more buildbot restart). On the console view it will stay green. But.. this is not a toy. Flaky tests are bad. We should mark tests flaky only if we really have to, and if you do, please make sure to file a P1 bug. Set the owner of the bug to whoever regressed the test. If you can't find who regressed the test, assign it to the person who originally wrote the test. Once we start tagging the flaky tests, we will monitor the flakiness dashboard and make sure that a test that is no longer flaky has its FLAKY_ tag removed. Let me know if you have questions. Thanks Nicolas --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
