Re: [DISCUSS] Releasable trunk and quality

[Joshua McKenzie]

A wise man once said "Simple is a feature" ;)

Our current process (commit oldest, merge up or merge -s ours w/ --amend):
- is confusing for new contributors to understand
- hides changes inside the merge commit
- masks future ability to see things with git attribute on commits
- is exposed to race w/other committers across multiple branches requiring
--atomic
- is non-automatable requiring human intervention and prone to error
- prevents us from using industry standard tooling and workflows around CI
thus contributing to CI degrading over time
+ Helps enforce that we don't forget to apply something to all branches
+(?) Is the devil we know

That's a lot of negatives for a very fixable single positive and some FUD.

On Tue, Jan 4, 2022 at 7:01 PM bened...@apache.org 
wrote:

> To answer your point, I don’t have anything ideologically against a
> temporary divergence in treatment, but we should have a clear unified
> endpoint we are aiming for.
>
>
>
> I would hate for this discussion to end without a clear answer about what
> that endpoint should be, though - even if we don’t get there immediately.
>
>
>
> I personally dislike the idea of relying on scripts to enforce this, at
> least in the long run, as there is no uniformity of environment, so no
> uniformity of process, and when things go wrong due to diverging systems
> we’re creating additional work for people (and CI is headache enough when
> it goes wrong).
>
>
>
>
>
> *From: *bened...@apache.org 
> *Date: *Tuesday, 4 January 2022 at 23:52
> *To: *David Capwell , Joshua McKenzie <
> jmcken...@apache.org>
> *Cc: *Henrik Ingo , dev <
> dev@cassandra.apache.org>
> *Subject: *Re: [DISCUSS] Releasable trunk and quality
>
> That all sounds terribly complicated to me.
>
>
>
> My view is that we should switch to the branch strategy outlined by Henrik
> (I happen to prefer it anyway) and move to GitHub integrations to control
> merge for each branch independently. Simples.
>
>
>
>
>
> *From: *David Capwell 
> *Date: *Tuesday, 4 January 2022 at 23:33
> *To: *Joshua McKenzie 
> *Cc: *Henrik Ingo , dev <
> dev@cassandra.apache.org>
> *Subject: *Re: [DISCUSS] Releasable trunk and quality
>
> The more I think on it, the more I am anyway strongly -1 on having some
> bifurcated commit process. We should decide on a uniform commit process for
> the whole project, for all patches, whatever that may be.
>
>
>
> Making the process stable and handle all the random things we need to
> handle takes a lot of time, for that reason I strongly feel we should start
> with trunk only and look to expand to other branches and/or handle
> multi-branch commits.  I agree that each branch should NOT have a different
> process, but feel its ok if we are evolving what the process should be.
>
>
>
> About the merge commit thing, we can automate (think Josh wants to OSS my
> script) the current process so this isn’t a blocker for automation; the
> thing I hate about it is that I have not found any tool able to understand
> our history, so it forces me to go to CLI to figure out how the merge
> actually changed things (only the smallest version can be displayed
> properly), I am 100% in favor of removing, but don’t think its a dependency
> on automating our merge process.
>
>
>
>
>
>
> On Jan 4, 2022, at 11:58 AM, Joshua McKenzie  wrote:
>
>
>
> I put together a draft confluence wiki page (login required) for the Build
> Lead role covering what we discussed in the thread here. Link:
> https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199527692&draftShareId=96dfa1ef-d927-427a-bff8-0cf711c790c9&;
>
>
>
> The only potentially controversial thing in there is text under what to do
> with a consistent test failure introduced by a diff to trunk: "If
> consistent, *git revert* *the SHA that introduced the failure*, re-open
> the original JIRA ticket, and leave a note for the original assignee about
> the breakage they introduced".
>
>
>
> This would apply only to patches to trunk that introduce consistent
> failures to a test clearly attributable to that patch.
>
>
>
> I am deferring on the topic of merge strategy as there's a lot of progress
> we can make without considering that more controversial topic yet.
>
>
>
> On Tue, Dec 21, 2021 at 9:02 AM Henrik Ingo 
> wrote:
>
> FWIW, I thought I could link to an example MongoDB commit:
>
>
>
>
> https://github.com/mongodb/mongo/commit/dec388494b652488259072cf61fd987af3fa8470
>
>
>
> * Fixes start from trunk or whatever is the highest version that includes
> the bug
>
> * It is then cherry picked to each stable version that needs to fix. Above
> link is an example of such a cherry pick. The original sha is referenced in
> the commit message.
>
> * I found that it makes sense to always cherry pick from the immediate
> higher version, since if you had to make some changes to the previous
> commit, they probably need to be in the next one as well.
>
> * There are no merge commits. Everything is always cherry picked or
> rebased to the

Re: [DISCUSS] Releasable trunk and quality

[Joshua McKenzie]

 > A wise man once said "Simple is a feature" ;)
For those unfamiliar, this was a reference to a sentiment Jonathan has
expressed over the years which I strongly agree with
https://issues.apache.org/jira/browse/CASSANDRA-6809?focusedCommentId=14102901&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14102901


On Wed, Jan 5, 2022 at 9:52 AM Joshua McKenzie  wrote:

> A wise man once said "Simple is a feature" ;)
>
> Our current process (commit oldest, merge up or merge -s ours w/ --amend):
> - is confusing for new contributors to understand
> - hides changes inside the merge commit
> - masks future ability to see things with git attribute on commits
> - is exposed to race w/other committers across multiple branches requiring
> --atomic
> - is non-automatable requiring human intervention and prone to error
> - prevents us from using industry standard tooling and workflows around CI
> thus contributing to CI degrading over time
> + Helps enforce that we don't forget to apply something to all branches
> +(?) Is the devil we know
>
> That's a lot of negatives for a very fixable single positive and some FUD.
>
> On Tue, Jan 4, 2022 at 7:01 PM bened...@apache.org 
> wrote:
>
>> To answer your point, I don’t have anything ideologically against a
>> temporary divergence in treatment, but we should have a clear unified
>> endpoint we are aiming for.
>>
>>
>>
>> I would hate for this discussion to end without a clear answer about what
>> that endpoint should be, though - even if we don’t get there immediately.
>>
>>
>>
>> I personally dislike the idea of relying on scripts to enforce this, at
>> least in the long run, as there is no uniformity of environment, so no
>> uniformity of process, and when things go wrong due to diverging systems
>> we’re creating additional work for people (and CI is headache enough when
>> it goes wrong).
>>
>>
>>
>>
>>
>> *From: *bened...@apache.org 
>> *Date: *Tuesday, 4 January 2022 at 23:52
>> *To: *David Capwell , Joshua McKenzie <
>> jmcken...@apache.org>
>> *Cc: *Henrik Ingo , dev <
>> dev@cassandra.apache.org>
>> *Subject: *Re: [DISCUSS] Releasable trunk and quality
>>
>> That all sounds terribly complicated to me.
>>
>>
>>
>> My view is that we should switch to the branch strategy outlined by
>> Henrik (I happen to prefer it anyway) and move to GitHub integrations to
>> control merge for each branch independently. Simples.
>>
>>
>>
>>
>>
>> *From: *David Capwell 
>> *Date: *Tuesday, 4 January 2022 at 23:33
>> *To: *Joshua McKenzie 
>> *Cc: *Henrik Ingo , dev <
>> dev@cassandra.apache.org>
>> *Subject: *Re: [DISCUSS] Releasable trunk and quality
>>
>> The more I think on it, the more I am anyway strongly -1 on having some
>> bifurcated commit process. We should decide on a uniform commit process for
>> the whole project, for all patches, whatever that may be.
>>
>>
>>
>> Making the process stable and handle all the random things we need to
>> handle takes a lot of time, for that reason I strongly feel we should start
>> with trunk only and look to expand to other branches and/or handle
>> multi-branch commits.  I agree that each branch should NOT have a different
>> process, but feel its ok if we are evolving what the process should be.
>>
>>
>>
>> About the merge commit thing, we can automate (think Josh wants to OSS my
>> script) the current process so this isn’t a blocker for automation; the
>> thing I hate about it is that I have not found any tool able to understand
>> our history, so it forces me to go to CLI to figure out how the merge
>> actually changed things (only the smallest version can be displayed
>> properly), I am 100% in favor of removing, but don’t think its a dependency
>> on automating our merge process.
>>
>>
>>
>>
>>
>>
>> On Jan 4, 2022, at 11:58 AM, Joshua McKenzie 
>> wrote:
>>
>>
>>
>> I put together a draft confluence wiki page (login required) for the
>> Build Lead role covering what we discussed in the thread here. Link:
>> https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199527692&draftShareId=96dfa1ef-d927-427a-bff8-0cf711c790c9&;
>>
>>
>>
>> The only potentially controversial thing in there is text under what to
>> do with a consistent test failure introduced by a diff to trunk: "If
>> consistent, *git revert* *the SHA that introduced the failure*, re-open
>> the original JIRA ticket, and leave a note for the original assignee about
>> the breakage they introduced".
>>
>>
>>
>> This would apply only to patches to trunk that introduce consistent
>> failures to a test clearly attributable to that patch.
>>
>>
>>
>> I am deferring on the topic of merge strategy as there's a lot of
>> progress we can make without considering that more controversial topic yet.
>>
>>
>>
>> On Tue, Dec 21, 2021 at 9:02 AM Henrik Ingo 
>> wrote:
>>
>> FWIW, I thought I could link to an example MongoDB commit:
>>
>>
>>
>>
>> https://github.com/mongodb/mongo/commit/dec388494b652488259072cf61fd987af3fa8470
>>
>>
>>
>> * Fixe

Re: Clarifying CI release criteria

[Joshua McKenzie]

Could you note that in the wiki in the comments section Ekaterina?
https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199530280&draftShareId=7c72c252-918c-456b-9859-7d12e8fa9309&;

Thanks!


On Tue, Jan 4, 2022 at 4:21 PM Ekaterina Dimitrova 
wrote:

> Hey Josh,
> Thank you for leading these discussions and organizing the wiki pages
> (also from the other mail).
> I just wanted to mention about point 4 of Pending work - I have a draft
> version for CircleCI usage, also Andres has updated the rst documents
> around running tests in a loop, etc.
> BUT those are pending on the merge of the ascii docs (to convert his work)
> and add the correct links and any other changes that have happened since I
> wrote it last year. Just saying for awareness before anyone decides to make
> a write up on that not knowing a draft already exists.
>
> Thanks,
> Ekaterina
>
> On Tue, 4 Jan 2022 at 14:51, Joshua McKenzie  wrote:
>
>> Here's a link to a draft article for the confluence wiki covering what we
>> discussed on this thread:
>> https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199530280&draftShareId=7c72c252-918c-456b-9859-7d12e8fa9309&;
>>
>> Assuming this article accurately captures what we discussed here as well
>> as outstanding work, I'll get it published and integrated with the
>> confluence wiki and get the remainder of the work into a JIRA epic to be
>> tracked.
>>
>> On Fri, Dec 17, 2021 at 4:41 PM Joshua McKenzie 
>> wrote:
>>
>>> I'll get this into a draft article on the wiki so we can collab on those
>>> 3 outstanding TBD's without further cluttering up the dev list. :)
>>>
>>> On Fri, Dec 17, 2021 at 11:38 AM Ekaterina Dimitrova <
>>> e.dimitr...@gmail.com> wrote:
>>>
 It’s indeed good call but I thought this will be addressed in a separate
 document where we discuss required test suites to be run pre-commit. If
 not
 - then I guess we should add those things here too?

 On Fri, 17 Dec 2021 at 11:36, Joshua McKenzie 
 wrote:

 > Good call; thanks for the reminder.
 >
 > So maybe add a
 >
 > 3.a: Run all new or modified tests through either local or remote
 > multiplexer N (TBD - 50?) times (w/link to instructions, etc)
 > 3.b Non-regressing is defined here...
 > 3.c After merging tickets...
 >
 > On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams 
 > wrote:
 >
 > > Could we also add something about running new tests through the
 > > multiplexer?
 > >
 > > On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <
 jmcken...@apache.org>
 > > wrote:
 > > >
 > > > So to clarify it all in one place, the proposed new CI process we
 > should
 > > > test for consensus is:
 > > >
 > > > 1. Canonical CI for a release is ci-cassandra. We can optionally,
 and
 > in
 > > > practice will, run circle as well but don't codify blocking on
 that.
 > > > 2. (NEW) We don't release unless we get a fully green run.
 > > > 3. Before any merge, you need either a non-regressing (i.e. no new
 > > > failures) run of circleci with a (specific suite of tests TBD) or
 of
 > > > ci-cassandra.
 > > >  3.a Non-regressing is defined here as "Doesn't introduce any
 new
 > > test
 > > > failures; any new failures in CI are clearly not attributable to
 this
 > > diff"
 > > >  3.b: (NEW) After merging tickets, ci-cassandra runs against
 the
 > SHA
 > > > and the author gets an advisory update on the related JIRA for
 any new
 > > > errors on CI. The author of the ticket will take point on
 triaging this
 > > new
 > > > failure and either fixing (if clearly reproducible or related to
 their
 > > > work) or opening a JIRA for the intermittent failure and linking
 it in
 > > > butler (https://butler.cassandra.apache.org/#/)
 > > > 4. (NEW) The Build Lead role + Butler catches and documents all
 > failures
 > > > and anything that slips through the procedural cracks in 3.b;
 > resourcing
 > > > for fixing flakey tests TBD
 > > >
 > > > Our two TBD we can tackle separately from consensus on the above:
 > > > 1. Suite of tests on circle required to be considered ready for
 merge
 > > > 2. How we resource fixing flakey tests that are functionally
 impossible
 > > to
 > > > attribute without essentially fixing the flake
 > > >
 > > > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <
 > > e.dimitr...@gmail.com>
 > > > wrote:
 > > >
 > > > > +1 (nb) on my end too, I second Mick
 > > > > Thanks for putting this together Josh
 > > > >
 > > > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever 
 > wrote:
 > > > >
 > > > > > >
 > > > > > >
 > > > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already
 do
 > > this)
 > > > > and
 > > > > > > get an advi

Re: [DISCUSS] Releasable trunk and quality

[Mick Semb Wever]

> - hides changes inside the merge commit
>


This is what all merge commits do. If you see a merge commit in the
history, isn't it normal to presume that it will contain the additional
change for that branch for the parent commit getting merged in?



> - is exposed to race w/other committers across multiple branches requiring
> --atomic
>


This is a positive, IMHO.
Folk forget to pull, rebase, then go to push and realise one of their
patches on a branch needs rebasing and rework. That rework may make them
reconsider the patch going to the other branches too.



> +(?) Is the devil we know
>

+ Bi-directional relationship between patches showing which branches it was
applied to (and how).  From the original commit or any of the merge commits
I can see which branches, and where the original commit, was applied. (See
the mongo example from Henrik, how do I see which other branches the trunk
commit was committed to? do i have to start text searching the git history
or going through the ticket system :-(

+ Developing patch on hardest branch first, then working on each softer
branch. I don't know how important this is, but I have found it a useful
practice that encourages smaller, more precise patches overall.


Agree we want a defined end goal. And all for experimenting and testing
simpler/cleaner approaches.




>

Re: [DISCUSS] Releasable trunk and quality

[bened...@apache.org]

> If you see a merge commit in the history, isn't it normal to presume that it 
> will contain the additional change for that branch for the parent commit 
> getting merged in?

Sure, but it is exceptionally non-trivial to treat the work as a single diff in 
any standard UX. In practice it becomes 3 or 4 diffs, none of which tell the 
whole story (and all of which bleed legacy details of earlier branches). This 
is a genuine significant pain point when doing archaeology, something I and 
others do quite frequently, and a large part of why I want to see them gone.

> Folk forget to pull, rebase, then go to push and realise one of their patches 
> on a branch needs rebasing and rework. That rework may make them reconsider 
> the patch going to the other branches too.

Conversely, this is exceptionally painful when maintaining branches and forks, 
and I can attest to the pain of maintaining these branches so they may be 
committed atomically to having wasted literal person-weeks in my time on the 
project. I do not recall experiencing a significant benefit in return.

> do i have to start text searching the git history

Yes. This is so simple as to be a non-issue - surely you must search git log 
semi-regularly? It is a frequent part of the job of developing against the 
project in my experience.

> Developing patch on hardest branch first, then working on each softer branch. 
> I don't know how important this is, but I have found it a useful practice 
> that encourages smaller, more precise patches overall.

I don’t think this strategy determines which branch is first developed against. 
However, if it did, it would seem to me to be a clear mark against the current 
system, which incentivises fully developing against the oldest version before 
forward-porting its entirety. Developing primarily against the most recent 
branch incentivises back-porting more minimal versions of the work, once the 
scope of the work is fully understood.

Re: [DISCUSS] Releasable trunk and quality

[bened...@apache.org]

I think simple, consistent, reliable and unavoidable are *the* killer features 
for QA. All features (give or take) of the industry standard approach of using 
CI hooks to gate PR merge.

From: Joshua McKenzie 
Date: Wednesday, 5 January 2022 at 14:53
To: dev 
Subject: Re: [DISCUSS] Releasable trunk and quality
A wise man once said "Simple is a feature" ;)

Our current process (commit oldest, merge up or merge -s ours w/ --amend):
- is confusing for new contributors to understand
- hides changes inside the merge commit
- masks future ability to see things with git attribute on commits
- is exposed to race w/other committers across multiple branches requiring 
--atomic
- is non-automatable requiring human intervention and prone to error
- prevents us from using industry standard tooling and workflows around CI thus 
contributing to CI degrading over time
+ Helps enforce that we don't forget to apply something to all branches
+(?) Is the devil we know

That's a lot of negatives for a very fixable single positive and some FUD.

On Tue, Jan 4, 2022 at 7:01 PM bened...@apache.org 
mailto:bened...@apache.org>> wrote:
To answer your point, I don’t have anything ideologically against a temporary 
divergence in treatment, but we should have a clear unified endpoint we are 
aiming for.

I would hate for this discussion to end without a clear answer about what that 
endpoint should be, though - even if we don’t get there immediately.

I personally dislike the idea of relying on scripts to enforce this, at least 
in the long run, as there is no uniformity of environment, so no uniformity of 
process, and when things go wrong due to diverging systems we’re creating 
additional work for people (and CI is headache enough when it goes wrong).


From: bened...@apache.org 
mailto:bened...@apache.org>>
Date: Tuesday, 4 January 2022 at 23:52
To: David Capwell mailto:dcapw...@apple.com>>, Joshua 
McKenzie mailto:jmcken...@apache.org>>
Cc: Henrik Ingo mailto:henrik.i...@datastax.com>>, 
dev mailto:dev@cassandra.apache.org>>
Subject: Re: [DISCUSS] Releasable trunk and quality
That all sounds terribly complicated to me.

My view is that we should switch to the branch strategy outlined by Henrik (I 
happen to prefer it anyway) and move to GitHub integrations to control merge 
for each branch independently. Simples.


From: David Capwell mailto:dcapw...@apple.com>>
Date: Tuesday, 4 January 2022 at 23:33
To: Joshua McKenzie mailto:jmcken...@apache.org>>
Cc: Henrik Ingo mailto:henrik.i...@datastax.com>>, 
dev mailto:dev@cassandra.apache.org>>
Subject: Re: [DISCUSS] Releasable trunk and quality
The more I think on it, the more I am anyway strongly -1 on having some 
bifurcated commit process. We should decide on a uniform commit process for the 
whole project, for all patches, whatever that may be.

Making the process stable and handle all the random things we need to handle 
takes a lot of time, for that reason I strongly feel we should start with trunk 
only and look to expand to other branches and/or handle multi-branch commits.  
I agree that each branch should NOT have a different process, but feel its ok 
if we are evolving what the process should be.

About the merge commit thing, we can automate (think Josh wants to OSS my 
script) the current process so this isn’t a blocker for automation; the thing I 
hate about it is that I have not found any tool able to understand our history, 
so it forces me to go to CLI to figure out how the merge actually changed 
things (only the smallest version can be displayed properly), I am 100% in 
favor of removing, but don’t think its a dependency on automating our merge 
process.



On Jan 4, 2022, at 11:58 AM, Joshua McKenzie 
mailto:jmcken...@apache.org>> wrote:

I put together a draft confluence wiki page (login required) for the Build Lead 
role covering what we discussed in the thread here. Link: 
https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199527692&draftShareId=96dfa1ef-d927-427a-bff8-0cf711c790c9&;

The only potentially controversial thing in there is text under what to do with 
a consistent test failure introduced by a diff to trunk: "If consistent, git 
revert the SHA that introduced the failure, re-open the original JIRA ticket, 
and leave a note for the original assignee about the breakage they introduced".

This would apply only to patches to trunk that introduce consistent failures to 
a test clearly attributable to that patch.

I am deferring on the topic of merge strategy as there's a lot of progress we 
can make without considering that more controversial topic yet.

On Tue, Dec 21, 2021 at 9:02 AM Henrik Ingo 
mailto:henrik.i...@datastax.com>> wrote:
FWIW, I thought I could link to an example MongoDB commit:

https://github.com/mongodb/mongo/commit/dec388494b652488259072cf61fd987af3fa8470

* Fixes start from trunk or whatever is the highest version that includes the 
bug
* It is then cherry pic