With that said, I guess we can just revise on a regular basis what exactly > are the last flakes and not numbers which also change quickly up and down > with the first change in the Infra. >
+1, I am in favour of taking a pragmatic approach. If flakies are identified and triaged enough that, with correlation from both CI systems, we are confident that no legit bugs are behind them, I'm in favour of going beta. I still remain in favour of somehow incentivising reducing other flakies as well. Flakies that expose poor/limited CI infra, and/or tests that are not as resilient as they could be, are still noise that indirectly reduce our QA (and increase efforts to find and tackle those legit runtime problems). Interested in hearing input from others here that have been spending a lot of time on this front. Could it work if we say: all flakies must be ticketed, and test/infra related flakies do not block a beta release so long as there are fewer than the previous release? The intent here being pragmatic, but keeping us on a "keep the campground cleaner" trajectory…
