Re: [Proposal] add pull request template

[Amit Aggarwal]

Hi Erick, 

If you are the same Erick, Thank you for answering lots of questions related to 
Cassandra on stack overflow. 


Thanks
Amit Aggarwal 

> On 16-Aug-2022, at 2:28 PM, Erick Ramirez  wrote:
> 
> I forgot to mention that whatever gets decided, I think it should be 
> implemented across all GitHub repos of the project. See here for the complete 
> list -- https://projects.apache.org/committee.html?cassandra 
> . Cheers!


-- 
**The content of this e-mail is confidential and is intended solely for the 
use of the individual or entity to whom it is addressed. If you have 
received this e-mail by mistake, please reply to this e-mail and follow 
with its deletion. If you are not the intended recipient, please note that 
it shall be considered unlawful to copy, forward or in any manner reveal 
the contents of this e-mail or any part thereof to anyone. Although 
Freshworks has taken reasonable precautions to ensure no malware is present 
in this e-mail, Freshworks cannot accept responsibility for any loss or 
damage arising from the use of this e-mail or attachments.**

Cassandra project status update 2022-08-17

[Josh McKenzie]

This update comes to you from day 5 of quarantining in the basement. Thanks 
Pandemic. (╯°□°)╯︵ ┻━┻

(Today we're going to test if the ASF mailing lists allows a variety of ascii 
characters! I almost hope for everyone's sakes it doesn't; I abuse these 
things. :))

Let's start with 4.1:
Latest run has 7 failures. If we dig a bit deeper into the detail panel 
(https://butler.cassandra.apache.org/#/ci/upstream/compare/Cassandra-4.1/cassandra-4.1),
 you can see that the CASTest failures in 
https://issues.apache.org/jira/browse/CASSANDRA-17461 account for the long pole 
blocking the release. Looks like there's multiple folks working on that (thanks 
Brandon, Benedict, Andres, and Berenguer!), but it also looks like there's 
still no assignee so we're maybe holding it at arms length. Either that or 
we're just going to keep dogpiling on it which is great too; I don't see it 
falling off the radar any time soon.

testAutoSnapshotTTIOnDropAfterRestart has failed a few times so there's some 
legit flake there: 
https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.distributed.test/AutoSnapshotTtlTest/testAutoSnapshotTTlOnDropAfterRestart_2/.
 No build lead lately so we don't have a JIRA for it or associated with it 
(https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252);
 I may put that mantle back on in the near future.

There are 2 other failures that push us up to 7:
1) 
org.apache.cassandra.distributed.test.RepairTest.testForcedNormalRepairWithOneNodeDown
 
(https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.distributed.test/RepairTest/testForcedNormalRepairWithOneNodeDown/).
 Looks like not all endpoints replied to the repair request so probably worth 
trying to repro locally and troubleshoot.

2) org.apache.cassandra.net.ProxyHandlerConnectionsTest.testExpireSome 
(https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.net/ProxyHandlerConnectionsTest/testExpireSome_2/).
 This is a timeout, so it's anyone's guess. :)

Holistically, if we take a step back and look at 4.1 from a distance as to its 
general CI health, there's quite a bit of flake there: 
https://butler.cassandra.apache.org/#/ci/upstream/compare/Cassandra-4.1/cassandra-4.1.
 If we toss out build 122 as an anomaly, there's many that fail once in the 
past 16 runs. This continues to highlight for me the tradeoff of re-running 
flakes vs. blocking on them. We've chatted a bit on other ML threads on this; 
while prepping this email I just noticed that the high level dashboard on our 
branches shows varied and pervasive flakiness which is particularly challenging.

Getting to 0 flakes with a "run once 0 tolerance" policy with the current ASF 
CI infra (which is definitely being improved upon!) looks to be something of a 
Sisyphean task. 

We're down from 13 tickets blocking 4.1 beta down to 7: 
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
 As mentioned above, we have some test failures w/out tickets so that 7 is 
probably closer realistically to the previous count.

We have one unassigned ticket blocking 4.1 if anyone wants to pick it up: 
https://issues.apache.org/jira/browse/CASSANDRA-17773 (Incorrect 
cassandra.logdir on Debian systems).


[New Contributors Getting Started]
Follow your curiosity! We have a small number of things that still need to be 
fixed blocking 4.1, but if you have something specific you're interested in and 
there's an open ticket in jira on Cassandra, feel free to ping in slack (see 
below) to see if there's any context you need to dive in and get yourself 
assigned to that ticket. Hit up the @cassandra_mentors alias to reach 
volunteers who are available to help you get situated and link up with you as a 
mentor.

To search JIRA for a topic of interest, replace "ReplaceTextHere" with the 
topic on the following JIRA search: 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20cassandra%20AND%20resolution%20!%3D%20unresolved%20AND%20assignee%20is%20EMPTY%20AND%20summary%20~%20%27ReplaceTextHere%27%20ORDER%20BY%20priority%20ASC

To get situated, here's an explanation of various types of contribution: 
https://cassandra.apache.org/_/community.html#how-to-contribute
An overview of the C* architecture: 
https://cassandra.apache.org/doc/latest/cassandra/architecture/overview.html
And here's our getting started contributing guide: 
https://cassandra.apache.org/_/development/index.html
We hang out in #cassandra-dev on https://the-asf.slack.com so come join us.


[Dev list Digest]
https://lists.apache.org/list?dev@cassandra.apache.org:lte=2w:

We've had a fairly active couple of weeks. Caleb is shopping for feedback on 
what we do with hints during decommission: 
https://lists.apache.org/thread/0o2kd2hntbdjhpf8t1j9l9ys7k7y1wo5. See 
CASSANDRA-17808 for more details: 
https://issues.apache.org/jira/browse/CASSANDRA-17808

Claude brought up the state of our 

Re: Cassandra project status update 2022-08-17

[Ekaterina Dimitrova]

One correction, testAutoSnapshotTTIOnDropAfterRestart - ticket in review
already, it wasn’t linked in Butler though. I will link it now. Thanks
Paulo and Caleb for looking into it.

On Wed, 17 Aug 2022 at 14:05, Josh McKenzie  wrote:

> This update comes to you from day 5 of quarantining in the basement.
> Thanks Pandemic. (╯°□°)╯︵ ┻━┻
>
> (Today we're going to test if the ASF mailing lists allows a variety of
> ascii characters! I almost hope for everyone's sakes it doesn't; I abuse
> these things. :))
>
> Let's start with 4.1:
> Latest run has 7 failures. If we dig a bit deeper into the detail panel (
> https://butler.cassandra.apache.org/#/ci/upstream/compare/Cassandra-4.1/cassandra-4.1),
> you can see that the CASTest failures in
> https://issues.apache.org/jira/browse/CASSANDRA-17461 account for the
> long pole blocking the release. Looks like there's multiple folks working
> on that (thanks Brandon, Benedict, Andres, and Berenguer!), but it also
> looks like there's still no assignee so we're maybe holding it at arms
> length. Either that or we're just going to keep dogpiling on it which is
> great too; I don't see it falling off the radar any time soon.
>
> has failed a few times so there's some legit flake there:
> https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.distributed.test/AutoSnapshotTtlTest/testAutoSnapshotTTlOnDropAfterRestart_2/.
> No build lead lately so we don't have a JIRA for it or associated with it (
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252);
> I may put that mantle back on in the near future.
>
> There are 2 other failures that push us up to 7:
> 1)
> org.apache.cassandra.distributed.test.RepairTest.testForcedNormalRepairWithOneNodeDown
> (
> https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.distributed.test/RepairTest/testForcedNormalRepairWithOneNodeDown/).
> Looks like not all endpoints replied to the repair request so probably
> worth trying to repro locally and troubleshoot.
>
> 2) org.apache.cassandra.net.ProxyHandlerConnectionsTest.testExpireSome (
> https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.net/ProxyHandlerConnectionsTest/testExpireSome_2/).
> This is a timeout, so it's anyone's guess. :)
>
> Holistically, if we take a step back and look at 4.1 from a distance as to
> its general CI health, there's quite a bit of flake there:
> https://butler.cassandra.apache.org/#/ci/upstream/compare/Cassandra-4.1/cassandra-4.1.
> If we toss out build 122 as an anomaly, there's many that fail once in the
> past 16 runs. This continues to highlight for me the tradeoff of re-running
> flakes vs. blocking on them. We've chatted a bit on other ML threads on
> this; while prepping this email I just noticed that the high level
> dashboard on our branches shows varied and pervasive flakiness which is
> particularly challenging.
>
> Getting to 0 flakes with a "run once 0 tolerance" policy with the current
> ASF CI infra (which is definitely being improved upon!) looks to be
> something of a Sisyphean task.
>
> We're down from 13 tickets blocking 4.1 beta down to 7:
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
> As mentioned above, we have some test failures w/out tickets so that 7 is
> probably closer realistically to the previous count.
>
> We have one unassigned ticket blocking 4.1 if anyone wants to pick it up:
> https://issues.apache.org/jira/browse/CASSANDRA-17773 (Incorrect
> cassandra.logdir on Debian systems).
>
>
> [New Contributors Getting Started]
> Follow your curiosity! We have a small number of things that still need to
> be fixed blocking 4.1, but if you have something specific you're interested
> in and there's an open ticket in jira on Cassandra, feel free to ping in
> slack (see below) to see if there's any context you need to dive in and get
> yourself assigned to that ticket. Hit up the @cassandra_mentors alias to
> reach volunteers who are available to help you get situated and link up
> with you as a mentor.
>
> To search JIRA for a topic of interest, replace "ReplaceTextHere" with the
> topic on the following JIRA search:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20cassandra%20AND%20resolution%20!%3D%20unresolved%20AND%20assignee%20is%20EMPTY%20AND%20summary%20~%20%27ReplaceTextHere%27%20ORDER%20BY%20priority%20ASC
>
> To get situated, here's an explanation of various types of contribution:
> https://cassandra.apache.org/_/community.html#how-to-contribute
> An overview of the C* architecture:
> https://cassandra.apache.org/doc/latest/cassandra/architecture/overview.html
> And here's our getting started contributing guide:
> https://cassandra.apache.org/_/development/index.html
> We hang out in #cassandra-dev on https://the-asf.slack.com so come join
> us.
>
>
> [Dev list Digest]
> https://lists.apache.org/list?dev@cassandra.apache.org:lte=2w:
>
> We've ha

[Marketing] For Review: Changelog #18

[Chris Thornett]

Hello everyone,

He's the monthly Changelog blog for July. As usual, this has a 72-hr
community review by lazy consensus.

Please add any amends and suggestions in the comments:
https://docs.google.com/document/d/1g7HlGHWDYJ0eOtpM5x0fnCyXQdI0XgRMiN1zrZyqK-k/edit?usp=sharing


Publication will be ASAP, as I'm catching back up with the usual content
pipeline cadence.

FYI - We also have a Deep Dive on CommitLog, thanks to Alex Sorokoumov,
that I'll be pushing through as soon as I can.

Thanks,

-- 

Chris Thornett
Senior Content Strategist, Constantia.io

Re: Cassandra project status update 2022-08-17

[Mick Semb Wever]

>
> We're down from 13 tickets blocking 4.1 beta down to 7:
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
> As mentioned above, we have some test failures w/out tickets so that 7 is
> probably closer realistically to the previous count.
>


I suggest we move to beta when all non-flaky-test tickets are resolved and
we get our first green ci-cassandra run.
And I suggest we move to rc when we get three consecutive green runs.

We did something similar last time, this would be the same exception to the
rules, rules we continue to get closer to.

An alternative is to replace "green" with "builds with only non-regression
and infra-caused failures".



> - It's pretty expensive and painful to defer cleaning up CI to the end of
> the release cycle
>


This^

Re: Cassandra project status update 2022-08-17

[Paulo Motta]

> testAutoSnapshotTTIOnDropAfterRestart has failed a few times so there's
some legit flake there:
https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.distributed.test/AutoSnapshotTtlTest/testAutoSnapshotTTlOnDropAfterRestart_2/.
No build lead lately so we don't have a JIRA for it or associated with it (
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252);
I may put that mantle back on in the near future.

There's https://issues.apache.org/jira/browse/CASSANDRA-17804, for some
reason it's not showing up in the kanban board..

Em qua., 17 de ago. de 2022 às 17:24, Mick Semb Wever 
escreveu:

> We're down from 13 tickets blocking 4.1 beta down to 7:
>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
>> As mentioned above, we have some test failures w/out tickets so that 7 is
>> probably closer realistically to the previous count.
>>
>
>
> I suggest we move to beta when all non-flaky-test tickets are resolved and
> we get our first green ci-cassandra run.
> And I suggest we move to rc when we get three consecutive green runs.
>
> We did something similar last time, this would be the same exception to
> the rules, rules we continue to get closer to.
>
> An alternative is to replace "green" with "builds with only non-regression
> and infra-caused failures".
>
>
>
>> - It's pretty expensive and painful to defer cleaning up CI to the end of
>> the release cycle
>>
>
>
> This^
>

Re: Cassandra project status update 2022-08-17

[Paulo Motta]

Please disconsider previous message, I missed Ekaterina's message :)

Em qua., 17 de ago. de 2022 às 17:35, Paulo Motta 
escreveu:

> > testAutoSnapshotTTIOnDropAfterRestart has failed a few times so there's
> some legit flake there:
> https://ci-cassandra.apache.org/job/Cassandra-4.1/138/testReport/org.apache.cassandra.distributed.test/AutoSnapshotTtlTest/testAutoSnapshotTTlOnDropAfterRestart_2/.
> No build lead lately so we don't have a JIRA for it or associated with it (
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252);
> I may put that mantle back on in the near future.
>
> There's https://issues.apache.org/jira/browse/CASSANDRA-17804, for some
> reason it's not showing up in the kanban board..
>
> Em qua., 17 de ago. de 2022 às 17:24, Mick Semb Wever 
> escreveu:
>
>> We're down from 13 tickets blocking 4.1 beta down to 7:
>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
>>> As mentioned above, we have some test failures w/out tickets so that 7 is
>>> probably closer realistically to the previous count.
>>>
>>
>>
>> I suggest we move to beta when all non-flaky-test tickets are resolved
>> and we get our first green ci-cassandra run.
>> And I suggest we move to rc when we get three consecutive green runs.
>>
>> We did something similar last time, this would be the same exception to
>> the rules, rules we continue to get closer to.
>>
>> An alternative is to replace "green" with "builds with only
>> non-regression and infra-caused failures".
>>
>>
>>
>>> - It's pretty expensive and painful to defer cleaning up CI to the end
>>> of the release cycle
>>>
>>
>>
>> This^
>>
>

Re: Cassandra project status update 2022-08-17

[Ekaterina Dimitrova]

+1, I second Mick on both points.

On Wed, 17 Aug 2022 at 16:23, Mick Semb Wever  wrote:

> We're down from 13 tickets blocking 4.1 beta down to 7:
>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455.
>> As mentioned above, we have some test failures w/out tickets so that 7 is
>> probably closer realistically to the previous count.
>>
>
>
> I suggest we move to beta when all non-flaky-test tickets are resolved and
> we get our first green ci-cassandra run.
> And I suggest we move to rc when we get three consecutive green runs.
>
> We did something similar last time, this would be the same exception to
> the rules, rules we continue to get closer to.
>
> An alternative is to replace "green" with "builds with only non-regression
> and infra-caused failures".
>
>
>
>> - It's pretty expensive and painful to defer cleaning up CI to the end of
>> the release cycle
>>
>
>
> This^
>

Re: Cassandra project status update 2022-08-17

[Berenguer Blasi]

+1 to Mick's points.

Also notice in circle 4.1 green runs are the norm lately imo. Yes it's 
not the official CI but it helps build an overall picture of improvement 
towards green CI. On jenkins, if you check the latest 4.1 runs, <5-ish 
failures per run are starting to be common and those that don't are 
known failures being worked on (CAS i.e.), infra or flakies taking you 
back to the <5-ish failures. So overall, if I am not missing anything, 
the signal among the infra and flaky noise is pretty good.


Regards

On 17/8/22 22:50, Ekaterina Dimitrova wrote:

+1, I second Mick on both points.

On Wed, 17 Aug 2022 at 16:23, Mick Semb Wever  wrote:

We're down from 13 tickets blocking 4.1 beta down to 7:

https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455

.
As mentioned above, we have some test failures w/out tickets
so that 7 is probably closer realistically to the previous count.



I suggest we move to beta when all non-flaky-test tickets are
resolved and we get our first green ci-cassandra run.
And I suggest we move to rc when we get three consecutive green runs.

We did something similar last time, this would be the same
exception to the rules, rules we continue to get closer to.

An alternative is to replace "green" with "builds with only
non-regression and infra-caused failures".

- It's pretty expensive and painful to defer cleaning up CI to
the end of the release cycle



This^

Re: [Proposal] add pull request template

[Erick Ramirez]

Yes, it's me. Guilty as charged. 😁

I appreciate the acknowledgement. Cheers!

On Thu, 18 Aug 2022 at 03:00, Amit Aggarwal 
wrote:

> Hi Erick,
>
> If you are the same Erick, Thank you for answering lots of questions
> related to Cassandra on stack overflow.
>
>
> Thanks
> Amit Aggarwal
>
> On 16-Aug-2022, at 2:28 PM, Erick Ramirez 
> wrote:
>
> I forgot to mention that whatever gets decided, I think it should be
> implemented across all GitHub repos of the project. See here for the
> complete list -- https://projects.apache.org/committee.html?cassandra.
> Cheers!
>
>
>
> *The content of this e-mail is confidential and is intended solely for the
> use of the individual or entity to whom it is addressed. If you have
> received this e-mail by mistake, please reply to this e-mail and follow
> with its deletion. If you are not the intended recipient, please note that
> it shall be considered unlawful to copy, forward or in any manner reveal
> the contents of this e-mail or any part thereof to anyone. Although
> Freshworks has taken reasonable precautions to ensure no malware is present
> in this e-mail, Freshworks cannot accept responsibility for any loss or
> damage arising from the use of this e-mail or attachments.*

Re: [Proposal] add pull request template

[Benedict]

Yes, I strongly endorse long form commit messages, though I’m often remiss 
about them myself. Until IDEs auto cross-reference JIRA, and we start 
sanitising our JIRA summaries, a commit record is going to be a better 
description of what has been done and why.

I think Jon is another proponent, who is more consistent about practicing what 
he preaches. It would be great to formalise this as an expectation, as if we’re 
honest the cost is not high to write a paragraph or two distilling the work 
that was done.

> On 16 Aug 2022, at 17:36, Josh McKenzie  wrote:
> 
> 
>> 
>> something more descriptive like "Describe what the pull request fixes, why 
>> it is needed, and what branch(s) you expect it to be applied to."
> I recall us having a conversation a long while ago about our commit messages 
> and whether they're optimal in their current format (defer all context to 
> JIRA) or whether there's value in adding a longer form digest of context in a 
> paragraph below the commit.
> 
> Over time I've become more sympathetic to the approach of informative 
> long-form bodies (I think you advocated for this in the past Benedict?) - 
> google does a decent job of documenting the process and reasoning here: 
> https://google.github.io/eng-practices/review/developer/cl-descriptions.html
> 
> One of the benefits of including information about the context of what you're 
> doing in the commit message, even if it's redundant with the JIRA ticket, is 
> that someone in-IDE using integrated annotation/blame can get the "why" of a 
> code-change in their existing workflow without having to jump out to a JIRA 
> ticket where the signal/noise ratio is often much poorer. This efficiency or 
> inefficiency is magnified the more historical patches you need to understand 
> to understand the context of the change you're making.
> 
> Changing our commit message process to include this would also translate into 
> having that information available in PR's, and in the PR description body, 
> which we could easily lift over to the commit itself and/or JIRA ticket.
> 

Re: [Proposal] add pull request template

[Benedict]

> By submitting this pull request, I acknowledge that I am making a 
> contribution to the Apache Software Foundation under the terms and conditions 
> of the [Contributor's 
> Agreement](https://www.apache.org/licenses/contributor-agreements.html).

Do we expect every contributor who makes any contribution to agree to this? If 
we don’t, does it imply anything weird to start stipulating this for PRs only? 
Might be worth running past legal, as we don’t expect ICLAs to be signed for 
many contributions today, nor any specific assignment besides the notice at the 
top of any files in the contribution.
I’m also not convinced we should be doing much more than letting the user know 
they should have read and followed the style guide and contribution guide, and 
that before being merged a commit should include documentation and test changes 
- but these aren’t probably blockers for an initial review, and might 
discourage contributions by making the initial hurdles appear higher.

> On 16 Aug 2022, at 09:01, Claude Warren, Jr via dev 
>  wrote:
> 
> 
> I am all for simplification.  How about
> 
> - start of text 
> 
> Issue resolved:  CASSANDRA-
> 
> 
>  - [ ] Jira ticket contains a description of: what is fixed, why it is 
> needed, and what branches to apply it to.
>  - [ ] Commits have been squashed to remove intermediate development commit 
> messages.
>  - [ ] Key commit messages start with the issue number (CASSANDRA-)
> 
> either
>  - [ ] this is a trivial documentation change. (e.g. fixes a typo)
> or:
>  - [ ] Tests are included.
>  - [ ] Documentation changes and/or updates are included.
>  
> 
> By submitting this pull request, I acknowledge that I am making a 
> contribution to the Apache Software Foundation under the terms and conditions 
> of the [Contributor's 
> Agreement](https://www.apache.org/licenses/contributor-agreements.html).
> 
> See the [Apache Cassandra "Contributing to Cassandra" 
> guide](https://cassandra.apache.org/_/development/index.html) and/or the 
> [Apache Cassandra "Working on Documentation" 
> guide](https://cassandra.apache.org/_/development/documentation.html)
> 
>  
>  end of text 
> 
>> On Tue, Aug 16, 2022 at 8:42 AM Erick Ramirez  
>> wrote:
>> +1 this is a great idea. But personally, I'm not too fussed about the level 
>> of detail that is in the template -- what is important is that contributors 
>> are reminded that there needs to be a ticket associated with contributions. 
>> Without being too prescriptive, aspiring contributors should really 
>> familiarise themselves with how to contribute[1] so they would know to 
>> search existing tickets first to avoid things like duplication.
>> 
>>  Additionally, I personally prefer details about a contribution to be 
>> documented in a ticket rather than a PR because information stored in 
>> tickets are more persistent. Having said that, it doesn't hurt to have the 
>> details included in the PR as long as it is in the ticket too. Cheers!