Ehsan Akhgari wrote: > Hi everyone, > > As part of our efforts to get more value out of the Talos test suite for > preventing performance regressions, we believe that we are now ready to put > a first set of measures against startup time regressions. We will start by > imposing a new backout policy for mozilla-inbound checkins for regressions > more than 4% on any given platform. If your patch falls in a range which > causes more than 4% Ts regression, it will be backed out by our sheriffs > together with the rest of the patches in that range, and you can only > reland after you fix the regression by testing locally or on the try server. > > The 4% threshold has been chosen based on anecdotal evidence on the most > recent Ts regressions that we have seen, and is too generous, but we will > be working to improve the reporting and regression detection systems > better, and as those get improved, we would feel more comfortable with > imposing this policy on other Talos tests with tighter thresholds. > > Please let me know if you have any questions. >
Do we still have the bug where a test that finishes first, but is from a later cset (say a later cset IMPROVES Ts by 4% or more) would make us think we regressed it on an earlier cset if that earlier talos run finishes later? Such that we set graph points by the time the test finished, not time the push was, etc. ~Justin Wood (Callek) _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
