I've witnessed PMCs -1 releases due to failing tests or bugs reported by users before, but prior to everyone's awesome work on CI I think a number of times folks might have been voting without knowing what the results of the full test runs were. One of the amazing contributions of this group (and others working on the CI/CD solutions over the years) is now we have an authoritative "which tests are failing" tool and I do hope we use it as context during the next release vote as suggested by this proposal.
I just think it should serve as context, and not a requirement. I also vote -1 on this specific proposal and will happily change it to +1 if the language on the release criteria is softened slightly, e.g. "When a release is proposed, links to the associated test runs on ci-cassandra.apache.org MUST be provided and members MAY use failing tests as a valid reason to -1 a release". -Joey On Wed, Jan 12, 2022 at 8:11 AM Ekaterina Dimitrova <e.dimitr...@gmail.com> wrote: > > “I particularly like the suggestion PMCs can use failing tests as a reason to > -1, but we do have critical patch releases now and again and common sense in > getting such releases out quickly needs to be applied. ” > > For some reason I assumed this would always be the case in case of emergency, > to consider it on a per case basis. Good catch on the wording! Thank you > Joeye! I think it doesn’t hurt to elaborate a bit more on this to be sure we > are all aligned that there will be special cases. (Hopefully not many) > > On Wed, 12 Jan 2022 at 3:25, Berenguer Blasi <berenguerbl...@gmail.com> wrote: >> >> Hi Joseph >> >> jenkins CI was at 2/3 flakies consistently post 4.0 release. Then things >> broke and we've been working hard to get back to the 2/3 flakies. Most >> current failures imo are timeuuid C17133 or early termination of process >> C17140 related afaik. So getting back to the 2/3 'impossible' flakies >> should be doable and a reasonable target (famous last words...). My 2cts. >> >> Regards >> >> On 12/1/22 7:21, Joseph Lynch wrote: >> > On Wed, Jan 12, 2022 at 12:47 AM Berenguer Blasi >> > <berenguerbl...@gmail.com> wrote: >> >> We shouldn't be at 15-20 failures but at 2 or 3. The problem is that >> >> those 2 or 3 have already been hammered for over a year by 2 or 3 >> >> different committers and they didn't crack. >> >> >> > Last I checked circleci was almost fully green on trunk only, and the >> > asf builds all had around 15-20 failures. For example, as of the last >> > build I checked, trunk had 22 failures [1], 4.0 had 12 [2], 3.11 had >> > 35 [3] and 3.0 had 25 [4]. >> > >> > [1] https://ci-cassandra.apache.org/job/Cassandra-trunk/901/ >> > [2] https://ci-cassandra.apache.org/job/Cassandra-4.0/308/ >> > [3] https://ci-cassandra.apache.org/job/Cassandra-3.11/300/ >> > [4] https://ci-cassandra.apache.org/job/Cassandra-3.0/234 >> > >> > Looking at the failures they mostly seem to be consistent failures >> > although there are some flakes as well. If I understand Josh's >> > proposal correctly, and I could be mistaken, but if this vote passes >> > it seems we would be unable to cut any release on any branch on the >> > project? >> > >> > -Joey >> > .
