I fully concede your point and concern Joey but I propose we phrase that differently to emphasize the importance of clean tests.
"All releases by default are expected to have a green test run on ci-cassandra Jenkins. In exceptional circumstances (security incidents, data loss, etc requiring hotfix), members with binding votes on a release may choose to approve a release with known failing tests." On Wed, Jan 12, 2022 at 11:36 AM Joseph Lynch <joe.e.ly...@gmail.com> wrote: > I've witnessed PMCs -1 releases due to failing tests or bugs reported > by users before, but prior to everyone's awesome work on CI I think a > number of times folks might have been voting without knowing what the > results of the full test runs were. One of the amazing contributions > of this group (and others working on the CI/CD solutions over the > years) is now we have an authoritative "which tests are failing" tool > and I do hope we use it as context during the next release vote as > suggested by this proposal. > > I just think it should serve as context, and not a requirement. I also > vote -1 on this specific proposal and will happily change it to +1 if > the language on the release criteria is softened slightly, e.g. "When > a release is proposed, links to the associated test runs on > ci-cassandra.apache.org MUST be provided and members MAY use failing > tests as a valid reason to -1 a release". > > -Joey > > On Wed, Jan 12, 2022 at 8:11 AM Ekaterina Dimitrova > <e.dimitr...@gmail.com> wrote: > > > > “I particularly like the suggestion PMCs can use failing tests as a > reason to -1, but we do have critical patch releases now and again and > common sense in getting such releases out quickly needs to be applied. ” > > > > For some reason I assumed this would always be the case in case of > emergency, to consider it on a per case basis. Good catch on the wording! > Thank you Joeye! I think it doesn’t hurt to elaborate a bit more on this to > be sure we are all aligned that there will be special cases. (Hopefully not > many) > > > > On Wed, 12 Jan 2022 at 3:25, Berenguer Blasi <berenguerbl...@gmail.com> > wrote: > >> > >> Hi Joseph > >> > >> jenkins CI was at 2/3 flakies consistently post 4.0 release. Then things > >> broke and we've been working hard to get back to the 2/3 flakies. Most > >> current failures imo are timeuuid C17133 or early termination of process > >> C17140 related afaik. So getting back to the 2/3 'impossible' flakies > >> should be doable and a reasonable target (famous last words...). My > 2cts. > >> > >> Regards > >> > >> On 12/1/22 7:21, Joseph Lynch wrote: > >> > On Wed, Jan 12, 2022 at 12:47 AM Berenguer Blasi > >> > <berenguerbl...@gmail.com> wrote: > >> >> We shouldn't be at 15-20 failures but at 2 or 3. The problem is that > those 2 or 3 have already been hammered for over a year by 2 or 3 different > committers and they didn't crack. > >> >> > >> > Last I checked circleci was almost fully green on trunk only, and the > >> > asf builds all had around 15-20 failures. For example, as of the last > >> > build I checked, trunk had 22 failures [1], 4.0 had 12 [2], 3.11 had > >> > 35 [3] and 3.0 had 25 [4]. > >> > > >> > [1] https://ci-cassandra.apache.org/job/Cassandra-trunk/901/ > >> > [2] https://ci-cassandra.apache.org/job/Cassandra-4.0/308/ > >> > [3] https://ci-cassandra.apache.org/job/Cassandra-3.11/300/ > >> > [4] https://ci-cassandra.apache.org/job/Cassandra-3.0/234 > >> > > >> > Looking at the failures they mostly seem to be consistent failures > >> > although there are some flakes as well. If I understand Josh's > >> > proposal correctly, and I could be mistaken, but if this vote passes > >> > it seems we would be unable to cut any release on any branch on the > >> > project? > >> > > >> > -Joey > >> > . >
