> Do we still have the bug where a test that finishes first, but is from a > later cset (say a later cset IMPROVES Ts by 4% or more) would make us > think we regressed it on an earlier cset if that earlier talos run > finishes later? > > Such that we set graph points by the time the test finished, not time > the push was, etc.
https://bugzilla.mozilla.org/show_bug.cgi?id=688534 On Thu, Oct 18, 2012 at 6:26 PM, Justin Wood (Callek) <cal...@gmail.com> wrote: > Ehsan Akhgari wrote: >> Hi everyone, >> >> As part of our efforts to get more value out of the Talos test suite for >> preventing performance regressions, we believe that we are now ready to put >> a first set of measures against startup time regressions. We will start by >> imposing a new backout policy for mozilla-inbound checkins for regressions >> more than 4% on any given platform. If your patch falls in a range which >> causes more than 4% Ts regression, it will be backed out by our sheriffs >> together with the rest of the patches in that range, and you can only >> reland after you fix the regression by testing locally or on the try server. >> >> The 4% threshold has been chosen based on anecdotal evidence on the most >> recent Ts regressions that we have seen, and is too generous, but we will >> be working to improve the reporting and regression detection systems >> better, and as those get improved, we would feel more comfortable with >> imposing this policy on other Talos tests with tighter thresholds. >> >> Please let me know if you have any questions. >> > > Do we still have the bug where a test that finishes first, but is from a > later cset (say a later cset IMPROVES Ts by 4% or more) would make us > think we regressed it on an earlier cset if that earlier talos run > finishes later? > > Such that we set graph points by the time the test finished, not time > the push was, etc. > > ~Justin Wood (Callek) > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
