A few thoughts:
1. If we gate the classification behind each test failure being root-caused,
we consistently need people who are dedicating their time to doing that or we
end up with a backlog of unclassified CI failures (like we have now and have
always had historically).
2. Newcomers to the
> I think a simple metric for "is something flaky" is "does it only fail once
> in the butler history (of 15 or so builds)".
Does that make it considered flaky? What if the one failure is a
timeout? I think each failing case has to have the failures
investigated in order to know.
Kind Regards,
So move to beta when:
1. all non-flaky test *failures* (NOT tickets, see below) are resolved
2. We get a green ci-cassandra run
Move to rc when:
1. Three consecutive green runs in ci-cassandra
Release when:
1. All rc tickets are closed
2. Some time-based gate maybe?
3. Three more consecutive
+1 to Mick's points.
Also notice in circle 4.1 green runs are the norm lately imo. Yes it's
not the official CI but it helps build an overall picture of improvement
towards green CI. On jenkins, if you check the latest 4.1 runs, <5-ish
failures per run are starting to be common and those that
+1, I second Mick on both points.
On Wed, 17 Aug 2022 at 16:23, Mick Semb Wever wrote:
> We're down from 13 tickets blocking 4.1 beta down to 7:
>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
>> As mentioned above, we have some test failures w/out ticket
Please disconsider previous message, I missed Ekaterina's message :)
Em qua., 17 de ago. de 2022 às 17:35, Paulo Motta
escreveu:
> > testAutoSnapshotTTIOnDropAfterRestart has failed a few times so there's
> some legit flake there:
> https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testRepor
> testAutoSnapshotTTIOnDropAfterRestart has failed a few times so there's
some legit flake there:
https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.distributed.test/AutoSnapshotTtlTest/testAutoSnapshotTTlOnDropAfterRestart_2/.
No build lead lately so we don't hav
>
> We're down from 13 tickets blocking 4.1 beta down to 7:
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
> As mentioned above, we have some test failures w/out tickets so that 7 is
> probably closer realistically to the previous count.
>
I suggest we mov
One correction, testAutoSnapshotTTIOnDropAfterRestart - ticket in review
already, it wasn’t linked in Butler though. I will link it now. Thanks
Paulo and Caleb for looking into it.
On Wed, 17 Aug 2022 at 14:05, Josh McKenzie wrote:
> This update comes to you from day 5 of quarantining in the bas
