Currently whenever a sheriff or developer classifies an intermittent failure on Treeherder with a bug number, a comment is left on that bug for every occurrence.
We're going to be turning these comments off, and replacing them with periodic summaries of recent failures, in order to: * Improve the signal to noise ratio of bug comments/bugmail. * Reduce the impact these comments have on Bugzilla (the current bot has made 670,000 comments over the years and the BMO team are understandably unimpressed by the bot!). * Move us one step closer to a crashstats-like model, where not all failures have bugs filed, and instead the canonical source of truth is a dashboard, from which the most actionable issues can be identified and prioritised. There will be two types of summary: a) Daily: Intended to warn of sudden spikes in failure rate, for which waiting a week would be too long. Posted to bugs with >= 15 failures/day across all trees. b) Weekly: The primary summary, intended to keep interested parties up to date and make it clear the bug is still occurring. Posted if >=5 failures/week across all trees. Using last week as an example - 8000 bug comments were left with the per-occurrence model across 800 bugs. With the above we would have instead only posted ~300 comments (for example content see bug 1179821 comment 13). The thresholds are an initial best estimate (see bug 1179821 comment 5) and will be tweaked in the future as needed. Longer term we have plans to: * Automatically classify failures where possible. * Use a "failure signature" as the canonical identifier for an intermittent failure, so we don't have to file bugs for every failure, only the most actionable. * Create an OrangeFactor v2 that uses Treeherder as the source of truth for intermittent failure data (as opposed to what is mirrored to ElasticSearch in a TBPL-era schema) and in so doing, give it the UX overhaul it has long needed. Barring any issues, this change will land this week. If you have any questions reply here or comment in bug 1179821 / bug 1179310, or see bug 1179263 for the longer term vision. Best wishes, Ed _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
