On Tue, Aug 4, 2015 at 9:14 AM, James Graham <ja...@hoppipolla.co.uk> wrote: > > 2) Start posting the results of test runs to treeherder, which will surface > the failures in a clearer way than the buildbot waterfall and will hopefully > in the near future get a measure of auto-matching test results against known > intermittents (this will probably be internal rather than using any > bugtracker as a backend, so it should be possible to also link specific > intermittents to github issues if you want that feature).
For those who aren't familiar with Firefox development, when you get an intermittent failure in Firefox tests (which is very frequent; it's rare to get a full test run that doesn't have at least one) treeherder will auto-suggest existing bugs that match the failure. This is *enormously* helpful for working out whether it's a known intermittent. It also requires that people are assiduous about filing bugs for intermittents. For Firefox the sheriffs do a great job of this. More generally, intermittent test failures are the thing that really stresses the Not Rocket Science Rule of Software Engineering (http://graydon.livejournal.com/186550.html). If you're writing a batch program like a compiler, it tends to work great. If you're writing a complex interactive app with lots of time-sensitive aspects, it gets harder. Nick _______________________________________________ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
