Re: [DISCUSSION] Flaky tests

2020-05-27 Thread Joshua McKenzie
> > So my idea was to suggest to start tracking an exact Jenkins report maybe? Basing our point of view on the canonical test runs on apache infra makes sense to me, assuming that infra is behaving these days. :) Pretty sure Mick got that in working order. At least for me, what I learned in the p

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Jordan West
On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie wrote: > Maybe. Do we just time box, say we're going to cut an RC and give it 4 > weeks, if nothing awful surfaces we GA? > I've seen that work well in the past on other projects. I agree with the notion that RCs are real candidates for release if

[DISCUSSION] Flaky tests

2020-05-27 Thread Ekaterina Dimitrova
Dear all, I spent some time these days looking into the Release Lifecycle document. As we keep on saying we approach Beta based on the Jira board, I was curious what is the exact borderline to cut it. Looking at all the latest reports (thanks to everyone who was working on that; I think having an

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Joshua McKenzie
Maybe. Do we just time box, say we're going to cut an RC and give it 4 weeks, if nothing awful surfaces we GA? On Wed, May 27, 2020 at 4:12 PM Brandon Williams wrote: > Absolutely my understanding. > > On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan > > wrote: > > > > A clear point to cut RC's

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Brandon Williams
Absolutely my understanding. On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan wrote: > > A clear point to cut RC's doesn't surface from the above for me. > Releasing > > an RC before broad verification seems wrong, and cutting an RC after the > 4 > > points above may as well be GA because it's al

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Jeremiah D Jordan
> A clear point to cut RC's doesn't surface from the above for me. Releasing > an RC before broad verification seems wrong, and cutting an RC after the 4 > points above may as well be GA because it's all known scope. Isn’t the whole point of an RC is that it could be the GA? It is a “release can

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Joshua McKenzie
I think we're all on the same page here; I was focusing more on the release lifecycles and sequencing than the entire version cycle. Good to broaden scope I think. One thing we're not considering is the separation of API changes from major changes and how that intersects with release milestones.

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Scott Andreas
That makes sense to me, yep. My hope and expectation is that the time required for "verification work" will shrink dramatically in the not too distant future - ideally to a period of less than a month. In this world, the cost of missing one train is reduced to catching the next one. One of the

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Jeremiah D Jordan
+1 strongly agree. If we aren’t going to let something go into 4.0.0 because it would "invalidate testing” then we can not let such a thing go into 4.0.1 unless we plan to re-do said testing for the patch release. > On May 27, 2020, at 1:31 PM, Benedict Elliott Smith > wrote: > > I'm being t

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Joshua McKenzie
In this hypothetical, certainly not post-ga, and I'd argue we shouldn't allow it post-beta1 and we need a clear demarcation of "this type of work is ok to merge before X, it's not ok after X. Validation testing *will not occur* before X, and will start after X". It's a bit rigid, but it's the only

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Benedict Elliott Smith
I'm being told this still isn't clear, so let me try in a bullet-point timeline: * 4.0 Beta * 4.0 Verification Work * [Merge Window] * 4.0 GA * 4.0 Minor Releases * ... * 5.0 Dev * ... * 5.0 Verification Work * GA 5.0 I think that anything that is prohibited from "[Merge Window]" because it in

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Benedict Elliott Smith
I'm not sure if I communicated my point very well. I mean to say that if the reason we are prohibiting a patch to land post-beta is because it invalidates work we only perform pre-ga, then it probably should not be permitted to land post-ga either, since it must also invalidate the same work?

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Joshua McKenzie
> > because it invalidates our pre-release verification, then it should not > land until we next perform pre-release verification At least for me there's a little softness around our collective alignment on when pre-release verification takes place. If it's between alpha-1 and ga we don't want ch

Fwd: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Ekaterina Dimitrova
Thank you all for your input. I think an important topic is again to revise the lifecycle and ensure we really have the vision on what is left until beta. I will start a separate thread on the flaky tests situation soon. For this particular ticket I see a couple of things: - There are a lot of del

Re: [DISCUSS] CASSANDRA-13994

2020-05-27 Thread Benedict Elliott Smith
I think our "pre-beta" criteria should also be our "not in a major" criteria. If work is prohibited because it invalidates our pre-release verification, then it should not land until we next perform pre-release verification, which only currently happens once per major. This could mean either l