What will be the policy if a test fails and it's unclear which push caused the regression? Is it the sheriff's job, or the people who pushed's job to figure out which push was the culprit and make sure that that push gets backed out?
I.e. if 4 pushes land between two testruns, and we see a regression, will the 4 pushes be backed out? Or will sheriffs run the missing tests and only back out the offending push? / Jonas On Thu, Aug 21, 2014 at 10:50 AM, Jonathan Griffin <jgrif...@mozilla.com> wrote: > Thanks Ed. To paraphrase, no test coverage is being lost here, we're just > being a little more deliberate with job coalescing. All tests will be run > on all platforms (including debug tests) on a commit before a merge to m-c. > > Jonathan > > > On 8/21/2014 9:35 AM, Ed Morley wrote: >> >> I think much of the pushback in this thread is due to a misunderstanding >> of some combination of: >> * our current buildbot scheduling >> * the proposal >> * how trees are sheriffed and merged >> >> To clarify: >> >> 1) We already have coalescing [*] of jobs on all trees apart from try. >> >> 2) This coalescing means that all jobs are still run at some point, but >> just may not run on every push. >> >> 3) When failures are detected, coalescing means that regression ranges are >> larger and so sometimes result in longer tree integration repo closures, >> whilst the sheriffs force trigger jobs on the revisions that did not >> originally run them. >> >> 4) When merging into mozilla-central, sheriffs ensure that all jobs are >> green - including those that got coalesced and those that are only scheduled >> periodically (eg non-unified & PGO builds are only run every 3 hours). (This >> is a fairly manual process currently, but better tooling should be possible >> with treeherder). >> >> 5) This proposal does not mean debug-only issues are somehow not worth >> acting on or that they'll end up shipped/on mozilla-central, thanks to #4. >> >> 6) This proposal is purely trying to make existing coalescing (#1/#2) more >> intelligent, to ensure that we expend the finite amount of machine time we >> have at present on the most appropriate jobs at each point, in order to >> reduce the impact of #3. >> >> Fwiw I'm on the fence as to whether the algorithm suggested in this >> proposal is the most effective way to aid with #3 - however it's worth >> trying to find out. >> >> Best wishes, >> >> Ed >> >> [*] Collapsing of pending jobs of the same type, when the queue size is >> greater than 1. > > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
