I recommend that instead of classifying intermittents as tests which fail > 30 times per week, to instead classify tests that fail more than some threshold percent as intermittent. Otherwise on a week with lots of checkins, a test which isn't actually a problem could clear the threshold and cause unnecessary work for orange triage people and developers alike.
The currently published threshold is 8%: https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy#Identifying_problematic_tests 8% seems reasonable to me. Also, whenever a test is disabled, not only should a bug be filed, but please _please_ need-info the test owner or at least someone on the affected team. If a test for a feature is disabled without the maintainer of that feature knowing, then we are flying blind and we are putting the quality of our product at risk. cpearce. On Wednesday, March 8, 2017 at 9:06:46 AM UTC+13, jma...@mozilla.com wrote: > On Tuesday, March 7, 2017 at 2:57:21 PM UTC-5, Steve Fink wrote: > > On 03/07/2017 11:34 AM, Joel Maher wrote: > > > Good suggestion here- I have seen so many cases where a simple > > > fix/disabled/unknown/needswork just do not describe it. Let me work on a > > > few new tags given that we have 248 bugs to date. > > > > > > I am thinking maybe [stockwell turnedoff] - where the job is turned off- > > > we > > > could also ensure one of the last comments indicates this. > > > > > > also [stockwell fix] -> [stockwell testfix], [stockwell bandaid] (for > > > those > > > requestLongerTimeouts(), etc.), [stockwell productfix], and [stockwell > > > reneabled]. > > > > Forgive the bikeshedding, but my kneejerk reaction to these is to wonder > > whether it's a good idea to use the "stockwell" jargon. It would be a > > lot easier for people unfamiliar with the stockwell project if these > > were [intermittent turnedoff], [intermittent fix], etc. Perhaps it's too > > late, but is that a possibility? > > I think that is valid, thanks for bringing that up! _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
