On Thursday 2013-11-07 14:13 +0200, Aryeh Gregor wrote: > On Wed, Nov 6, 2013 at 6:46 PM, Ryan VanderMeulen <rya...@gmail.com> wrote: > > I'm just afraid we're going to end up in the same situation we're already in > > with intermittent failures where the developer looks at it and says "that > > couldn't possibly be from me" and ignores it. We already see "Try results > > look good" backouts on a depressingly-regular basis. > > The entire situation with how intermittent failures are handled > strikes me as mostly a technical problem. Known intermittent failures > should be flagged and automatically suppressed, not require manual > judgment calls every single time. To ensure that they don't get made > non-intermittent, they could be automatically rerun a couple of times > (just the file, not the whole suite) if they fail to make sure they > pass at least once, and get reported as a real failure if they fail > five times in a row or something. Trying to persuade people to be > careful of something that isn't a problem 90% of the time is a losing > battle -- the signal-to-noise ratio needs to be a lot higher before > people will pay attention.
I think this depends on what you mean by "known intermittent failures". If a known intermittent failure is the result of any regression that leads to a previously-passing test failing intermittently, I'd be pretty uncomfortable with this. There have been quite a few JS engine changes that led to style system mochitests failing intermittently; I wouldn't want all of the style system's test coverage to be progressively turned off as a result. But if you're talking about new tests that aren't yet passing reliably, or other cases where the module owner of the test recognizes that the regression is acceptable, then that seems ok. We need to get better about identifying and backing out changes that cause previously-passing tests to start failing intermittently. This requires better tools for doing it. -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914)
signature.asc
Description: Digital signature
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
