Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

[Andrés de la Peña]

The proposal looks good to me. I have created tickets for:
 - Increasing the default number of repeated test iterations to 500 (
CASSANDRA-17937 ,
ready to commit)
 - Automatically detecting and repeating new or modified JUnit tests (
CASSANDRA-17939 ,
patch available)
 - Allowing to specify multiple tests in the test multiplexer (
CASSANDRA-17938 , in
progress)

On Mon, 3 Oct 2022 at 15:23, Josh McKenzie  wrote:

> Any further revisions or objections to this or are we good to take it to a
> vote?
>
> On Wed, Sep 28, 2022, at 10:54 AM, Josh McKenzie wrote:
>
> So revised proposal:
>
> On Release Lifecycle cwiki page:
>  - Ensure we have parity on jobs run between circle and asf-ci
>  - Allow usage of circleci as gatekeeper for releases. 1 green run ->
> beta, 3 green runs consecutive -> ga
>  - No new consistent regressions on CI for asf compared to prior branches
>  - Explicitly do not consider ci-cassandra asf flaky tests as release
> blockers
>
> Changes to codify into documentation:
>  - On patch before commit, multiplex @500 all new tests, changed tests, or
> expected to be impacted tests ("expected to be impacted" piece pending
> multi-class multiplexing support):
>  - Add support for multi-class specification in multiplexer and document
>
> Add informal project commitment during next major release lifecycle to
> continue working on bringing asf ci-cassandra up to where it can be formal
> gatekeeper for release.
>
> On Wed, Sep 28, 2022, at 10:13 AM, Ekaterina Dimitrova wrote:
>
> If we talk blockers nothing more than ensuring we see all tests we want
> pre-release, IMHO.
> The other points sound to me like future important improvements that will
> help us significantly in the flaky test fight.
>
> On Wed, 28 Sep 2022 at 10:08, Josh McKenzie  wrote:
>
>
> I'm receptive to that but I wouldn't gate our ability to get 4.1 out the
> door based on circle on that. Honestly probably only need to have the
> parity of coverage be the blocker for its use in retrospect.
>
> On Wed, Sep 28, 2022, at 1:32 AM, Berenguer Blasi wrote:
>
> I would add an option for generate.sh to detect all changed *Test.java
> files, that would be handy imo.
> On 28/9/22 4:29, Josh McKenzie wrote:
>
> So:
>
>1. 500 iterations on multiplexer
>2. Augmenting generate.sh to allow providing multiple class names and
>generating a single config that'll multiplex all the tests provided
>3. Test parity / pre-release config added on circleci (see
>https://issues.apache.org/jira/browse/CASSANDRA-17930),
>specifically dtest-large, dtest-offheap, test-large-novnode
>
> If we get the above 3, are we at a place where we're good to consider
> vetting releases on circleci for beta / rc / ga?
>
> On Tue, Sep 27, 2022, at 11:28 AM, Ekaterina Dimitrova wrote:
>
> “I have plans on modifying the multiplexer to allow specifying a list of
> classes per test target, so we don't have to needlessly suffer with this”
>
>
> That would be great, I was thinking of that the other day too. With that
> said I’ll be happy to support you in that effort too :-)
>
>
> On Tue, 27 Sep 2022 at 11:18, Josh McKenzie  wrote:
>
>
> I have plans on modifying the multiplexer to allow specifying a list of
> classes per test target, so we don't have to needlessly suffer with this
>
> This sounds integral to us multiplexing tests on large diffs whether we go
> with circle for releases or not and would be a great addition!
>
> On Tue, Sep 27, 2022, at 6:19 AM, Andrés de la Peña wrote:
>
> 250 iterations isn't enough; I use 500 as a low water mark.
>
>
> I agree that 500 iterations would be a reasonable minimum. We have seen
> flaky unit tests requiring far more iterations, but that's not very common.
> We could use to 500 iterations as default, and discretionary use a higher
> limit in tests that are quick and might be prone to concurrency issues. I
> can change the defaults on CirceCI config file if we agree to a new limit,
> the current default of 100 iterations is quite arbitrary.
>
> The test multiplexer allows to either run test individual test methods or
> entire classes. It is quite frequent to see tests methods that pass
> individually but fail when they are run together with the other tests in
> the same class. Because of this, I think that we should always run entire
> classes when repeating new or modified tests. The only exception to this
> would be Python dtests, which usually are more resource intensive and not
> so prone to that type of issues.
>
> For CI on a patch, run the pre-commit suite and also run multiplexer with
> 250 runs on new, changed, or related tests to ensure not flaky
>
>
> The multiplexer only allows to run a single test class per push. This is
> ok for fixing existing flakies (its original purpose), and for most minor
> changes, but it can be quite inconvenient fo

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

[Josh McKenzie]

Thanks Andres! Let's sit tight and wait for the release vote for beta-1 to 
settle before we ratify these changes.

On Wed, Oct 5, 2022, at 6:20 AM, Andrés de la Peña wrote:
> The proposal looks good to me. I have created tickets for:
>  - Increasing the default number of repeated test iterations to 500 
> (CASSANDRA-17937 , 
> ready to commit)
>  - Automatically detecting and repeating new or modified JUnit tests 
> (CASSANDRA-17939 , 
> patch available)
>  - Allowing to specify multiple tests in the test multiplexer 
> (CASSANDRA-17938 , in 
> progress)
> 
> On Mon, 3 Oct 2022 at 15:23, Josh McKenzie  wrote:
>> __
>> Any further revisions or objections to this or are we good to take it to a 
>> vote?
>> 
>> On Wed, Sep 28, 2022, at 10:54 AM, Josh McKenzie wrote:
>>> So revised proposal:
>>> 
>>> On Release Lifecycle cwiki page:
>>>  - Ensure we have parity on jobs run between circle and asf-ci
>>>  - Allow usage of circleci as gatekeeper for releases. 1 green run -> beta, 
>>> 3 green runs consecutive -> ga
>>>  - No new consistent regressions on CI for asf compared to prior branches
>>>  - Explicitly do not consider ci-cassandra asf flaky tests as release 
>>> blockers
>>> 
>>> Changes to codify into documentation:
>>>  - On patch before commit, multiplex @500 all new tests, changed tests, or 
>>> expected to be impacted tests ("expected to be impacted" piece pending 
>>> multi-class multiplexing support):
>>>  - Add support for multi-class specification in multiplexer and document
>>> 
>>> Add informal project commitment during next major release lifecycle to 
>>> continue working on bringing asf ci-cassandra up to where it can be formal 
>>> gatekeeper for release.
>>> 
>>> On Wed, Sep 28, 2022, at 10:13 AM, Ekaterina Dimitrova wrote:
 If we talk blockers nothing more than ensuring we see all tests we want 
 pre-release, IMHO. 
 The other points sound to me like future important improvements that will 
 help us significantly in the flaky test fight.
 
 On Wed, 28 Sep 2022 at 10:08, Josh McKenzie  wrote:
> __
> I'm receptive to that but I wouldn't gate our ability to get 4.1 out the 
> door based on circle on that. Honestly probably only need to have the 
> parity of coverage be the blocker for its use in retrospect.
> 
> On Wed, Sep 28, 2022, at 1:32 AM, Berenguer Blasi wrote:
>> I would add an option for generate.sh to detect all changed *Test.java 
>> files, that would be handy imo.
>> 
>> On 28/9/22 4:29, Josh McKenzie wrote:
>>> So:
>>>  1. 500 iterations on multiplexer
>>>  2. Augmenting generate.sh to allow providing multiple class names and 
>>> generating a single config that'll multiplex all the tests provided
>>>  3. Test parity / pre-release config added on circleci (see 
>>> https://issues.apache.org/jira/browse/CASSANDRA-17930), specifically 
>>> dtest-large, dtest-offheap, test-large-novnode
>>> If we get the above 3, are we at a place where we're good to consider 
>>> vetting releases on circleci for beta / rc / ga?
>>> 
>>> On Tue, Sep 27, 2022, at 11:28 AM, Ekaterina Dimitrova wrote:
> “I have plans on modifying the multiplexer to allow specifying a list 
> of classes per test target, so we don't have to needlessly suffer 
> with this”
> 
> 
> That would be great, I was thinking of that the other day too. With 
> that said I’ll be happy to support you in that effort too :-) 
 
 On Tue, 27 Sep 2022 at 11:18, Josh McKenzie  
 wrote:
> 
>> I have plans on modifying the multiplexer to allow specifying a list 
>> of classes per test target, so we don't have to needlessly suffer 
>> with this
> This sounds integral to us multiplexing tests on large diffs whether 
> we go with circle for releases or not and would be a great addition!
> 
> On Tue, Sep 27, 2022, at 6:19 AM, Andrés de la Peña wrote:
>>> 250 iterations isn't enough; I use 500 as a low water mark.
>> 
>> I agree that 500 iterations would be a reasonable minimum. We have 
>> seen flaky unit tests requiring far more iterations, but that's not 
>> very common. We could use to 500 iterations as default, and 
>> discretionary use a higher limit in tests that are quick and might 
>> be prone to concurrency issues. I can change the defaults on CirceCI 
>> config file if we agree to a new limit, the current default of 100 
>> iterations is quite arbitrary.
>> 
>> The test multiplexer allows to either run test individual test 
>> methods or entire classes. It is quite frequent to see tests methods 
>> that pass individually but

Re: [VOTE] Release Apache Cassandra 4.1-beta1

[Caleb Rackliffe]

+1

Also, FWIW, https://issues.apache.org/jira/browse/CASSANDRA-17927 has been
resolved.

On Mon, Oct 3, 2022 at 9:37 AM Ekaterina Dimitrova 
wrote:

> As long as those two are checked to be test issues and we agree to fix
> them before next release, I am also +1
>
> On Mon, 3 Oct 2022 at 9:42, Josh McKenzie  wrote:
>
>> +1
>>
>> On Mon, Oct 3, 2022, at 6:49 AM, Aleksey Yeshchenko wrote:
>>
>> +1
>>
>> > On 3 Oct 2022, at 06:27, Berenguer Blasi 
>> wrote:
>> >
>> > +1
>> >
>> > On 30/9/22 16:20, Brandon Williams wrote:
>> >> I'm +1 under these conditions as well.
>> >>
>> >> Kind Regards,
>> >> Brandon
>> >>
>> >> On Thu, Sep 29, 2022 at 3:34 PM Mick Semb Wever 
>> wrote:
>> 
>>  The vote will be open for 72 hours (longer if needed). Everyone who
>> has tested the build is invited to vote. Votes by PMC members are
>> considered binding. A vote passes if there are at least three binding +1s
>> and no -1's.
>> >>>
>> >>>
>> >>> So long as the ongoing 'release criteria' thread permits, I'm +1
>> >>> (I won't close the vote until that thread summarises, up to you Josh
>> when.)
>> >>>
>> >>> I'm ok with waivers on the two new flakies 17927 and 17928, so long
>> as they block the next (beta or rc) release. Both appear to be only test
>> related.
>> >>>
>> >>> Checked
>> >>> - signing correct
>> >>> - checksums are correct
>> >>> - source artefact builds
>> >>> - binary artefact runs
>> >>> - debian package runs
>> >>> - redhat package runs
>> >>>
>> >>>
>>
>>
>>

Re: [DISCUSS] Revising our release criteria, commit guidelines, and the role of circleci vs. ASF CI

[Mick Semb Wever]

> Thanks Andres! Let's sit tight and wait for the release vote for beta-1 to
> settle before we ratify these changes.
>


Reading this, I believe we have a lazy consensus in place and the 4.1-beta1
thread has settled, so we can finalise the release, while this thread will
go to ratification separately.

[RESULT][VOTE] Release Apache Cassandra 4.1-beta1

[Mick Semb Wever]

>
> Proposing the test build of Cassandra 4.1-beta1 for release.
>
> sha1: 5d9d93ea08d9c76402aa1d14bad54bf9ec875686
> Git:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.1-beta1-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1276/org/apache/cassandra/cassandra-all/4.1-beta1/
>
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/4.1-beta1/
>
> The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
>



 The vote passes with eleven +1's (five binding) and no vetoes.


Notes:
 - The next release is blocked on 17928.
 - The release goes ahead with agreed changes in release criteria (not yet
ratified), as described in
https://lists.apache.org/thread/y7kfdmsdrv6p1vvcsf397r276fypwvod

[RELEASE] Apache Cassandra 4.1-beta1 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 4.1-beta1.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is the first beta release[1] on the 4.1 series.

[WARNING] Debian and RedHat package repositories have moved! Debian
/etc/apt/sources.list.d/cassandra.sources.list and RedHat
/etc/yum.repos.d/cassandra.repo files must be updated to the new repository
URLs. For Debian it is now https://debian.cassandra.apache.org . For RedHat
it is now https://redhat.cassandra.apache.org/41x/ .

Enjoy!

[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.1-beta1
[2]: NEWS.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.1-beta1
[3]: https://issues.apache.org/jira/browse/CASSANDRA