The largest number of test failures turn out (as pointed out by David) to be due to how arcane it was to trigger the full test suite. Hopefully we can get on top of that, but I think a significant remaining issue is a lack of trust in the output of CI. It’s hard to gate commit on a clean CI run when there’s flaky tests, and it doesn’t take much to misattribute one failing test to the existing flakiness (I tend to compare to a run of the trunk baseline for comparison, but this is burdensome and still error prone). The more flaky tests there are the more likely this is.
This is in my opinion the real cost of flaky tests, and it’s probably worth trying to crack down on them hard if we can. It’s possible the Simulator may help here, when I finally finish it up, as we can port flaky tests to run with the Simulator and the failing seed can then be explored deterministically (all being well). From: Brandon Williams <dri...@gmail.com> Date: Wednesday, 3 November 2021 at 17:07 To: dev@cassandra.apache.org <dev@cassandra.apache.org> Subject: Re: [DISCUSS] Releasable trunk and quality On Mon, Nov 1, 2021 at 5:03 PM David Capwell <dcapw...@apple.com.invalid> wrote: > > > How do we define what "releasable trunk" means? > > One thing I would love is for us to adopt a “run all tests needed to release > before commit” mentality, and to link a successful run in JIRA when closing > (we talked about this once in slack). If we look at CircleCI we currently do > not run all the tests needed to sign off; below are the tests disabled in the > “pre-commit” workflows (see > https://github.com/apache/cassandra/blob/trunk/.circleci/config-2_1.yml#L381): A good first step toward this would be for us to treat our binding +1s more judiciously, and not grant any without at least a pre-commit CI run linked in the ticket. You don't have to look very hard to find a lot of these today (I know I'm guilty), and it's possible we wouldn't have the current CI mess now if we had been a little bit more diligent. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
