-------- Original-Nachricht --------
Betreff: Re: mozilla-inbound backout policy subject to change (become
similar to autoland)
Von: Boris Zbarsky <bzbar...@mit.edu>
Datum: 2018-06-24 21:28
On 6/19/18 9:04 AM, Sebastian Hengst wrote:
TL;DR: We would like to change the mozilla-inbound backout policy to
be like autoland’s.
This seems like a pretty reasonable change in general.
Is there a well-documented try syntax string which corresponds to "these
are the things that need to be green to avoid being backed out"?
Presumably that string is not "-p all -u all" because it should exclude
tier2 and lower things, but it's not entirely clear to me what the
autoland backout criteria are. I would assume we want developers to do
a try run with that syntax before pushing to inbound as needed.
-Boris
The recommend Try practices can be found at
https://wiki.mozilla.org/Sheriffing/How_To/Recommended_Try_Practices
Tier 1 needs to pass.
Tier 2 either needs to pass or (not be a mass failure and the developer
have an ETA for fix [up to 1 day]).
Tier 3 isn't watched by sheriffs.
Because new test suites and platforms start on a different platform than
tier 1, testing all jobs which are tier 1 and only those with a try
syntax is very hard. E.g. there are talos jobs in tier 1 and 2 and
Marionette on Android debug in tier 2 but as tier 1 on Android opt.
For the following usage groups (this doesn't cover everybody), this is
what I would recommend:
platform backend (no frontend like e.g. permission prompts for a website
permission, or hardware/heavily OS dependent features): Linux x64 both
opt and debug builds, pick the test suites you need (mochitest plain +
crashtest + xpcshell + web-platform-test + ...)
platform assuming it affects all platforms (with frontend like e.g.
permission prompts for a website permission, or hardware/heavily OS
dependent features): opt builds for Linux x64, macOS, Windows, Android;
Linux x64 debug builds, pick the test suites you need (mochitest plain +
crashtest + xpcshell + web-platform-test + ...)
frontend: linux64 opt, add macOS and Windows if it depends on platform
specific features, test mochitest browser-chrome and/or devtools,
mochitest clipboard (contains both browser-chrome and devtools tests
which perform better on beefier/our machines?), mochitest a11y,
marionette, xpcshell + ...
This a rough guideline and there is no syntax which will provide 100%
(some tier 1 jobs like localization only run on mozilla-central and not
the integration repositories).
If there is an issue with a patch which hasn't been spotted on Try
because it was e.g. intermittent, didn't affect every platform, was in a
test suite which focuses on testing something else or the patch
bitrotted, it will get backed out and be realistically regarded as
unavoidable.
The Try pushes are for a reasonable protection against cross-platform
permafails in test suites designed to test those changes. The faster
backouts shall keep inbound closed for shorter timeframes and backed out
patches shall be fixed locally and pushed again.
- Sebastian
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
