This is great. Thanks for this! Wes
On Mon, Sep 28, 2015 at 5:31 PM, Ed Morley <emor...@mozilla.com> wrote: > Currently whenever a sheriff or developer classifies an intermittent > failure on Treeherder with a bug number, a comment is left on that bug for > every occurrence. > > We're going to be turning these comments off, and replacing them with > periodic summaries of recent failures, in order to: > * Improve the signal to noise ratio of bug comments/bugmail. > * Reduce the impact these comments have on Bugzilla (the current bot has > made 670,000 comments over the years and the BMO team are understandably > unimpressed by the bot!). > * Move us one step closer to a crashstats-like model, where not all > failures have bugs filed, and instead the canonical source of truth is a > dashboard, from which the most actionable issues can be identified and > prioritised. > > There will be two types of summary: > a) Daily: Intended to warn of sudden spikes in failure rate, for which > waiting a week would be too long. Posted to bugs with >= 15 failures/day > across all trees. > b) Weekly: The primary summary, intended to keep interested parties up to > date and make it clear the bug is still occurring. Posted if >=5 > failures/week across all trees. > > Using last week as an example - 8000 bug comments were left with the > per-occurrence model across 800 bugs. With the above we would have instead > only posted ~300 comments (for example content see bug 1179821 comment 13). > The thresholds are an initial best estimate (see bug 1179821 comment 5) and > will be tweaked in the future as needed. > > Longer term we have plans to: > * Automatically classify failures where possible. > * Use a "failure signature" as the canonical identifier for an > intermittent failure, so we don't have to file bugs for every failure, only > the most actionable. > * Create an OrangeFactor v2 that uses Treeherder as the source of truth > for intermittent failure data (as opposed to what is mirrored to > ElasticSearch in a TBPL-era schema) and in so doing, give it the UX > overhaul it has long needed. > > Barring any issues, this change will land this week. If you have any > questions reply here or comment in bug 1179821 / bug 1179310, or see bug > 1179263 for the longer term vision. > > Best wishes, > > Ed > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
