Re: August 5.0 Freeze (with waivers…) and a 5.0-alpha1

[Miklosovic, Stefan]

I plan to run few builds with Corretto etc and do few manual tests, I will try 
to push that as soon as possible but it might happen it will leak to the next 
week.


From: Mick Semb Wever 
Sent: Thursday, July 27, 2023 0:27
To: dev
Subject: August 5.0 Freeze (with waivers…) and a 5.0-alpha1

NetApp Security WARNING: This is an external email. Do not click links or open 
attachments unless you recognize the sender and know the content is safe.




The previous thread¹ on when to freeze 5.0 landed on freezing the first week of 
August, with a waiver in place for TCM and Accord to land later (but before 
October).

With JDK8 now dropped and SAI and UCS merged, the only expected 5.0 work that 
hasn't landed is Vector search (CEP-30).

Are there any objections to a waiver on Vector search?  All the groundwork: SAI 
and the vector type; has been merged, with all remaining work expected to land 
in August.

I'm keen to freeze and see us shift gears – there's already SO MUCH in 5.0 and 
a long list of flakies.  It takes time and patience to triage and identify the 
bugs that hit us before GA.  The freeze is about being "mostly feature 
complete",  so we have room for things before our first beta (precedence is to 
ask).   If we hope for a GA by December, account for the 6 weeks turnaround 
time for cutting and voting on one alpha, one beta, and one rc release, and the 
quiet period that August is, we really only have September and October left.

I already feel this is asking a bit of a miracle from us given how 4.1 went 
(and I'm hoping I will be proven wrong).

In addition, are there any objections to cutting an 5.0-alpha1 release as soon 
as we freeze?

This is on the understanding vector, tcm and accord will become available in 
later alphas.  Originally the discussion¹ was waiting for Accord for alpha1, 
but a number of folk off-list have requested earlier alphas to help with 
testing.


¹) 
https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3

Re: [DISCUSS] Maintain backwards compatibility after dependency upgrade in the 5.0

[Maxim Muzafarov]

Bump this topic up for visibility as the code freeze is coming soon.

This seems like a good change to include in 5.0 as this kind of
library upgrade is more natural when the major version changes. It is
still possible to postpone it to 6.0, but the main concern here is
that the current version of dropwizard metrics library is obsolete and
no longer supported and it is better to avoid emergencies that could
arise (like the panic with log4j library upgrade some time ago).

The change itself is straightforward and deserves more eyes on it from
my point of view.

On Fri, 21 Jul 2023 at 14:51, Maxim Muzafarov  wrote:
>
> Hello everyone,
>
> It still needs a pair of eyes to push it forward.
>
>
> I came across another good thing that might help us to overcome the
> difficulties with the dropwizard metrics dependency upgrade. The
> change relates to the driver itself and reuses the same approach that
> was used to deal with the driver's netty dependencies. We need to
> shade the dropwizard metrics classes and no longer rely on the
> cassandra classpath at least for the 3.x version of the java driver,
> and make the next 3.11.4 release of the java driver accordingly.
>
> The changes for the driver are here:
> https://github.com/datastax/java-driver/pull/1685
>
> This will give us (and users as well) the confidence to move forward
> with this change to 5.x alongside the 3.11 version of the driver
> usage. Looking forward to your thoughts.
>
> Changes for the Cassandra part are here:
> https://github.com/apache/cassandra/pull/2238/files
>
> On Mon, 3 Jul 2023 at 15:15, Maxim Muzafarov  wrote:
> >
> > I'd like to mention the approach we took here: to untangle the driver
> > update in tests with the dropwizard library version (cassandra-driver
> > 3.11 requires the "old" JMXReporter classes in the classpath) we have
> > copied the classes into the tests themselves, as it is allowed by the
> > Apache License 2.0. This way we can update the metrics library itself
> > and then update the driver used in the tests afterwards.
> >
> > If there are no objections, we need another committer to take a look
> > at these changes:
> > https://issues.apache.org/jira/browse/CASSANDRA-14667
> > https://github.com/apache/cassandra/pull/2238/files
> >
> > Thanks in advance for your help!
> >
> > On Wed, 28 Jun 2023 at 16:04, Bowen Song via dev
> >  wrote:
> > >
> > > IMHO, anyone upgrading software between major versions should expect to
> > > see breaking changes. Introducing breaking or major changes is the whole
> > > point of bumping major version numbers.
> > >
> > > Since the library upgrade need to happen sooner or later, I don't see
> > > any reason why it should not happen in the 5.0 release.
> > >
> > >
> > > On 27/06/2023 19:21, Maxim Muzafarov wrote:
> > > > Hello everyone,
> > > >
> > > >
> > > > We use the Dropwizard Metrics 3.1.5 library, which provides a basic
> > > > set of classes to easily expose Cassandra internals to a user through
> > > > various interfaces (the most common being JMX). We want to upgrade
> > > > this library version in the next major release 5.0 up to the latest
> > > > stable 4.2.19 for the following reasons:
> > > > - the 3.x (and 4.0.x) Dropwizard Metrics library is no longer
> > > > supported, which means that if we face a critical CVE, we'll still
> > > > need to upgrade, so it's better to do it sooner and more calmly;
> > > > - as of 4.2.5 the library supports jdk11, jdk17, so we will be in-sync
> > > > [1] as well as having some of the compatibility fixes mentioned in the
> > > > related JIRA [2];
> > > > - there have been a few user-related requests [3][4] whose
> > > > applications collide with the old version of the library, we want to
> > > > help them;
> > > >
> > > >
> > > > The problem
> > > >
> > > > The problem with simply upgrading is that the JmxReporter class of the
> > > > library has moved from the com.codahale.metrics package in the 3.x
> > > > release to the com.codahale.metrics.jmx package in the 4.x release.
> > > > This is a problem for applications/tools that rely on the cassandra
> > > > classpath (lib/jars) as after the upgrade they may be looking for the
> > > > JmxReporter class which has changed its location.
> > > >
> > > > A good example of the problem that we (or a user) may face after the
> > > > upgrade is our tests and the cassandra-driver-core 3.1.1, which uses
> > > > the old 3.x version of the library in tests. Of course, in this case,
> > > > we can upgrade the cassandra driver up to 4.x [5][6] to fix the
> > > > problem, as the new driver uses a newer version of the library, but
> > > > that's another story I won't go into for now. I'm talking more about
> > > > visualising the problem a user might face after upgrading to 5.0 if
> > > > he/she rely on the cassandra classpath, but on the other hand, they
> > > > might not face this problem at all because, as I understand, they will
> > > > provide this library in their applications by themselves.
> > > >

Re: August 5.0 Freeze (with waivers…) and a 5.0-alpha1

[Ekaterina Dimitrova]

+1 on freeze 1st August with the suggested waiver.

Also, +1 on the idea for alpha 1, as long as we give straight upfront the
right expectations to users.

Now, when we added also JDK17 in CI, I want to encourage people to start
testing it in their test environments and give us some feedback.



On Wed, 26 Jul 2023 at 18:39, J. D. Jordan 
wrote:

> I think this plan seems reasonable to me. +1
>
> -Jeremiah
>
> On Jul 26, 2023, at 5:28 PM, Mick Semb Wever  wrote:
>
> 
>
>
> The previous thread¹ on when to freeze 5.0 landed on freezing the first
> week of August, with a waiver in place for TCM and Accord to land later
> (but before October).
>
> With JDK8 now dropped and SAI and UCS merged, the only expected 5.0 work
> that hasn't landed is Vector search (CEP-30).
>
> Are there any objections to a waiver on Vector search?  All the
> groundwork: SAI and the vector type; has been merged, with all remaining
> work expected to land in August.
>
> I'm keen to freeze and see us shift gears – there's already SO MUCH in 5.0
> and a long list of flakies.  It takes time and patience to triage and
> identify the bugs that hit us before GA.  The freeze is about being "mostly
> feature complete",  so we have room for things before our first beta
> (precedence is to ask).   If we hope for a GA by December, account for the
> 6 weeks turnaround time for cutting and voting on one alpha, one beta, and
> one rc release, and the quiet period that August is, we really only have
> September and October left.
>
> I already feel this is asking a bit of a miracle from us given how 4.1
> went (and I'm hoping I will be proven wrong).
>
> In addition, are there any objections to cutting an 5.0-alpha1 release as
> soon as we freeze?
>
> This is on the understanding vector, tcm and accord will become available
> in later alphas.  Originally the discussion¹ was waiting for Accord for
> alpha1, but a number of folk off-list have requested earlier alphas to help
> with testing.
>
>
> ¹) https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3
>
>

Re: [Discuss] Repair inside C*

[Josh McKenzie]

> The idea that your data integrity needs to be opt-in has never made sense to 
> me from the perspective of either the product or the end user.
I could not agree with this more. 100%.

> The current (and past) state of things where running the DB correctly 
> **requires* *running a separate process (either community maintained or 
> official C* sidecar) is incredibly painful for folks.
I'm 50/50 on this (and I have some opinions here; bear with me :D ).

To me this goes beyond the question of just "where do we coordinate repair" 
into "what role does a node play vs. the sidecar and how does that intersect 
w/the industry today".

Having just 1 process you run on N machines is much nicer from an operations 
standpoint and it's *much* cleaner and easier for us as a project to not have 
to deal with signaling, shmem, and going down the IPC rabbit hole. A modular 
monolith, if you will.

That said, I feel like zeitgeist has been all-in in terms of microservices and 
control planes, whether they're the right solution or not. The affordances on 
being able to build out independent teams and large organization dev velocity, 
never-mind the ideal of being able to cleanly upgrade or rewrite internal 
components, is attractive enough on paper that it feels like most groups have 
gone that direction and accepted the perceived costs; I view Cassandra as being 
something of an architectural anachronism at this point. And to call back to 
the prior paragraph, I *think* you get all those positive affordances w/a 
modular monolith. Sadly, google trends 

 don't really give me a lot of hope there.

In an ideal world operators (or better yet, an automated operations process) 
would be able to dynamically adjust resource allocation to nodes based on 
"burstiness of the buffering" (i.e. lots of data building up in CL's needing to 
be flushed, or compaction need, or repair need); It's not immediately obvious 
to me how we'd gracefully do that in a single process paradigm in containers 
w/out becoming a noisy neighbor but it's not impossible. Kind of goes meta 
outside C*'s scope into how you're coordinating your hardware and software 
interactions; maybe that's the cleaner route: we clearly signal metrics for 
each major operation the DB needs to do to indicate their backlog and an 
external orchestration process / system / ??? handles the resource allocation. 
i.e. we don't take that on.

Certainly we can do a lot better when it comes to internal scheduling of DB 
operations to one another than we are today (start using cql rate limiting, 
dynamically determine a rolling average of needs to smooth out burst requests, 
make byte-based rate-limiting an option, user-space threads w/loom and some 
kind of QoS prioritization based on backlogs, etc).

I personally view moving maintenance tasks into the sidecar as a reasonable 
"first step satisficing compromise". If anything, that'd potentially give us 
some breathing room to get our house in order on the "I/O" process (as opposed 
to sidecar as "maintenance process") to then re-integrate things back in in a 
more clean, planned fashion with some better tools to do it right.

~Josh


On Wed, Jul 26, 2023, at 7:20 PM, C. Scott Andreas wrote:
> I agree that it would be ideal for Cassandra to have a repair scheduler in-DB.
> 
> That said I would happily support an effort to bring repair scheduling to the 
> sidecar immediately. This has nothing blocking it, and would potentially 
> enable the sidecar to provide an official repair scheduling solution that is 
> compatible with current or even previous versions of the database.
> 
> Once TCM has landed, we’ll have much stronger primitives for repair 
> orchestration in the database itself. But I don’t think that should block 
> progress on a repair scheduling solution in the sidecar, and there is nothing 
> that would prevent someone from continuing to use a sidecar-based solution in 
> perpetuity if they preferred.
> 
> - Scott
> 
> > On Jul 26, 2023, at 3:25 PM, Jon Haddad  wrote:
> > 
> > I'm 100% in favor of repair being part of the core DB, not the sidecar.  
> > The current (and past) state of things where running the DB correctly 
> > *requires* running a separate process (either community maintained or 
> > official C* sidecar) is incredibly painful for folks.  The idea that your 
> > data integrity needs to be opt-in has never made sense to me from the 
> > perspective of either the product or the end user.
> > 
> > I've worked with way too many teams that have either configured this 
> > incorrectly or not at all.  
> > 
> > Ideally Cassandra would ship with repair built in and on by default.  Power 
> > users can disable if they want to continue to maintain their own repair 
> > tooling for some reason.
> > 
> > Jon
> > 
> >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
> >> All,
> >> We had a brief discussi

Re: [DISCUSS] Maintain backwards compatibility after dependency upgrade in the 5.0

[Josh McKenzie]

+1 to the change pre 5.0.

Any committers have bandwidth to review 
https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-14667?

PR can be found here: https://github.com/apache/cassandra/pull/2238/files

On Thu, Jul 27, 2023, at 7:59 AM, Maxim Muzafarov wrote:
> Bump this topic up for visibility as the code freeze is coming soon.
> 
> This seems like a good change to include in 5.0 as this kind of
> library upgrade is more natural when the major version changes. It is
> still possible to postpone it to 6.0, but the main concern here is
> that the current version of dropwizard metrics library is obsolete and
> no longer supported and it is better to avoid emergencies that could
> arise (like the panic with log4j library upgrade some time ago).
> 
> The change itself is straightforward and deserves more eyes on it from
> my point of view.
> 
> On Fri, 21 Jul 2023 at 14:51, Maxim Muzafarov  wrote:
> >
> > Hello everyone,
> >
> > It still needs a pair of eyes to push it forward.
> >
> >
> > I came across another good thing that might help us to overcome the
> > difficulties with the dropwizard metrics dependency upgrade. The
> > change relates to the driver itself and reuses the same approach that
> > was used to deal with the driver's netty dependencies. We need to
> > shade the dropwizard metrics classes and no longer rely on the
> > cassandra classpath at least for the 3.x version of the java driver,
> > and make the next 3.11.4 release of the java driver accordingly.
> >
> > The changes for the driver are here:
> > https://github.com/datastax/java-driver/pull/1685
> >
> > This will give us (and users as well) the confidence to move forward
> > with this change to 5.x alongside the 3.11 version of the driver
> > usage. Looking forward to your thoughts.
> >
> > Changes for the Cassandra part are here:
> > https://github.com/apache/cassandra/pull/2238/files
> >
> > On Mon, 3 Jul 2023 at 15:15, Maxim Muzafarov  wrote:
> > >
> > > I'd like to mention the approach we took here: to untangle the driver
> > > update in tests with the dropwizard library version (cassandra-driver
> > > 3.11 requires the "old" JMXReporter classes in the classpath) we have
> > > copied the classes into the tests themselves, as it is allowed by the
> > > Apache License 2.0. This way we can update the metrics library itself
> > > and then update the driver used in the tests afterwards.
> > >
> > > If there are no objections, we need another committer to take a look
> > > at these changes:
> > > https://issues.apache.org/jira/browse/CASSANDRA-14667
> > > https://github.com/apache/cassandra/pull/2238/files
> > >
> > > Thanks in advance for your help!
> > >
> > > On Wed, 28 Jun 2023 at 16:04, Bowen Song via dev
> > >  wrote:
> > > >
> > > > IMHO, anyone upgrading software between major versions should expect to
> > > > see breaking changes. Introducing breaking or major changes is the whole
> > > > point of bumping major version numbers.
> > > >
> > > > Since the library upgrade need to happen sooner or later, I don't see
> > > > any reason why it should not happen in the 5.0 release.
> > > >
> > > >
> > > > On 27/06/2023 19:21, Maxim Muzafarov wrote:
> > > > > Hello everyone,
> > > > >
> > > > >
> > > > > We use the Dropwizard Metrics 3.1.5 library, which provides a basic
> > > > > set of classes to easily expose Cassandra internals to a user through
> > > > > various interfaces (the most common being JMX). We want to upgrade
> > > > > this library version in the next major release 5.0 up to the latest
> > > > > stable 4.2.19 for the following reasons:
> > > > > - the 3.x (and 4.0.x) Dropwizard Metrics library is no longer
> > > > > supported, which means that if we face a critical CVE, we'll still
> > > > > need to upgrade, so it's better to do it sooner and more calmly;
> > > > > - as of 4.2.5 the library supports jdk11, jdk17, so we will be in-sync
> > > > > [1] as well as having some of the compatibility fixes mentioned in the
> > > > > related JIRA [2];
> > > > > - there have been a few user-related requests [3][4] whose
> > > > > applications collide with the old version of the library, we want to
> > > > > help them;
> > > > >
> > > > >
> > > > > The problem
> > > > >
> > > > > The problem with simply upgrading is that the JmxReporter class of the
> > > > > library has moved from the com.codahale.metrics package in the 3.x
> > > > > release to the com.codahale.metrics.jmx package in the 4.x release.
> > > > > This is a problem for applications/tools that rely on the cassandra
> > > > > classpath (lib/jars) as after the upgrade they may be looking for the
> > > > > JmxReporter class which has changed its location.
> > > > >
> > > > > A good example of the problem that we (or a user) may face after the
> > > > > upgrade is our tests and the cassandra-driver-core 3.1.1, which uses
> > > > > the old 3.x version of the library in tests. Of course, in this case,
> > > > > we can upgrade the cassandra driver up to 4.x [5][6] 

Re: August 5.0 Freeze (with waivers…) and a 5.0-alpha1

[Josh McKenzie]

+1 to what you've stated here Mick with a question: where did we land on 
flagging new features as experimental? Seems like it's an "at author's 
discretion" - search of the list turned up not too much structure there. Had a 
statement to that effect from Benjamin here 
.

I ask because I think there's *a lot* in 5.0 and it's not clear to me that the 
various features have been exercised in comparable ways to gain confidence in 
their prod readiness. For 5.0 we have (or will likely have):
 1. Trie Memtables
 2. Trie Indexed Tables
 3. UCS
 4. SAI
 5. Vector Search
 6. Transactional Metadata
 7. Accord
While 6 (TrM) can't fall under that umbrella for architectural reasons, for the 
rest the label might be useful.

Flip side, of course, is the argument that the features should be exercised 
heavily enough (unit, dtest, harry, simulator, in qa env, etc) before merge 
that nothing merges in that's not prod ready; don't know if that's true or not 
for all the features above (genuinely don't know; not looking to spread FUD), 
and it's not immediately obvious to me that that's the optimal way for us to 
balance innovation vs. stabilization.

Basically I think it'd be nice if "Experimental" fostered more safe innovation 
and expansion in our ecosystem rather than being something of a pariah dumping 
ground to retroactively apply to things that have proven unstable or not 
completed. :) Plus there'd be value in signaling that to users.

Hm. I think that implies feature categories of "alpha (experimental, may be 
removed (see MV's))", "beta (api stable and expected to be prod-hardened)", and 
then prod ready. Bit of a pandora's box now that I type this out; sorry.

On Wed, Jul 26, 2023, at 6:38 PM, J. D. Jordan wrote:
> 
> I think this plan seems reasonable to me. +1
> 
> -Jeremiah
> 
>> On Jul 26, 2023, at 5:28 PM, Mick Semb Wever  wrote:
>> 
>> 
>> The previous thread¹ on when to freeze 5.0 landed on freezing the first week 
>> of August, with a waiver in place for TCM and Accord to land later (but 
>> before October).
>> 
>> With JDK8 now dropped and SAI and UCS merged, the only expected 5.0 work 
>> that hasn't landed is Vector search (CEP-30).  
>> 
>> Are there any objections to a waiver on Vector search?  All the groundwork: 
>> SAI and the vector type; has been merged, with all remaining work expected 
>> to land in August.
>> 
>> I'm keen to freeze and see us shift gears – there's already SO MUCH in 5.0 
>> and a long list of flakies.  It takes time and patience to triage and 
>> identify the bugs that hit us before GA.  The freeze is about being "mostly 
>> feature complete",  so we have room for things before our first beta 
>> (precedence is to ask).   If we hope for a GA by December, account for the 6 
>> weeks turnaround time for cutting and voting on one alpha, one beta, and one 
>> rc release, and the quiet period that August is, we really only have 
>> September and October left.  
>> 
>> I already feel this is asking a bit of a miracle from us given how 4.1 went 
>> (and I'm hoping I will be proven wrong). 
>> 
>> In addition, are there any objections to cutting an 5.0-alpha1 release as 
>> soon as we freeze?  
>> 
>> This is on the understanding vector, tcm and accord will become available in 
>> later alphas.  Originally the discussion¹ was waiting for Accord for alpha1, 
>> but a number of folk off-list have requested earlier alphas to help with 
>> testing.
>> 
>> 
>> ¹) https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3

Re: [ANNOUNCEMENT] Expect failures today. Dropping JDK 8 and adding JDK 11

[Mick Semb Wever]

> I am working on dropping JDK 8 and adding JDK17 on trunk in both CI
systems today.
> This requires numerous patches in a few repos so you will be seeing more
failures in CI throughout the day today, but it shouldn’t be anything more
🤞 than what we have listed in the table of failures in CASSANDRA-16895’s
description. I will be applying the fixes one by one today.
> I will keep you posted with updates. Also, please, do let me know if you
have any questions or concerns.



FYI,  devbranch (pre-commit) on ci-cassandra.a.o is hardcoded to build on
jdk8 and jdk11
so it's not going to work with trunk based patches…

I'll review any patches to fix this, but otherwise focus is
on CASSANDRA-18594 where a standalone jenkinsfile avoids this problem
altogether.

Re: [ANNOUNCEMENT] Expect failures today. Dropping JDK 8 and adding JDK 11

[Ekaterina Dimitrova]

It seems there was some misunderstanding as dev-branch was on the list of
tasks to be fixed here.

Anyway, I will work on a patch as I do not want to block people who are
using it, thanks. My understanding from a private conversation is that
CASSANDRA-18594 might land in August-September so I don’t think we should
wait for it.
Apologize for any inconvenience caused, it was definitely not my intention

On Thu, 27 Jul 2023 at 12:49, Mick Semb Wever  wrote:

>
>
>
> > I am working on dropping JDK 8 and adding JDK17 on trunk in both CI
> systems today.
> > This requires numerous patches in a few repos so you will be seeing more
> failures in CI throughout the day today, but it shouldn’t be anything more
> 🤞 than what we have listed in the table of failures in CASSANDRA-16895’s
> description. I will be applying the fixes one by one today.
> > I will keep you posted with updates. Also, please, do let me know if you
> have any questions or concerns.
>
>
>
> FYI,  devbranch (pre-commit) on ci-cassandra.a.o is hardcoded to build on
> jdk8 and jdk11
> so it's not going to work with trunk based patches…
>
> I'll review any patches to fix this, but otherwise focus is
> on CASSANDRA-18594 where a standalone jenkinsfile avoids this problem
> altogether.
>
>

Re: [ANNOUNCEMENT] Expect failures today. Dropping JDK 8 and adding JDK 11

[Ekaterina Dimitrova]

CASSANDRA-18701

On Thu, 27 Jul 2023 at 13:10, Ekaterina Dimitrova 
wrote:

> It seems there was some misunderstanding as dev-branch was on the list of
> tasks to be fixed here.
>
> Anyway, I will work on a patch as I do not want to block people who are
> using it, thanks. My understanding from a private conversation is that
> CASSANDRA-18594 might land in August-September so I don’t think we should
> wait for it.
> Apologize for any inconvenience caused, it was definitely not my intention
>
> On Thu, 27 Jul 2023 at 12:49, Mick Semb Wever  wrote:
>
>>
>>
>>
>> > I am working on dropping JDK 8 and adding JDK17 on trunk in both CI
>> systems today.
>> > This requires numerous patches in a few repos so you will be seeing
>> more failures in CI throughout the day today, but it shouldn’t be anything
>> more 🤞 than what we have listed in the table of failures in
>> CASSANDRA-16895’s description. I will be applying the fixes one by one
>> today.
>> > I will keep you posted with updates. Also, please, do let me know if
>> you have any questions or concerns.
>>
>>
>>
>> FYI,  devbranch (pre-commit) on ci-cassandra.a.o is hardcoded to build on
>> jdk8 and jdk11
>> so it's not going to work with trunk based patches…
>>
>> I'll review any patches to fix this, but otherwise focus is
>> on CASSANDRA-18594 where a standalone jenkinsfile avoids this problem
>> altogether.
>>
>>

Re: Status Update on CEP-7 Storage Attached Indexes (SAI)

[Lorina Poland]

I have to say how excited I am that I get to emphasize SAI as an indexing
method for Apache C*, rather than the much less capable 2i! I have been
waiting for this day for years! Thanks Caleb and all who worked to make
this day a reality!

Lorina

On Wed, Jul 26, 2023 at 10:08 PM Berenguer Blasi 
wrote:

> Nice one!
> On 26/7/23 21:11, Ekaterina Dimitrova wrote:
>
> Thanks Caleb!
> Great  job everyone! 🚀👏🏻
>
> On Wed, 26 Jul 2023 at 15:07, J. D. Jordan 
> wrote:
>
>> Thanks for all the work here!
>>
>> On Jul 26, 2023, at 1:57 PM, Caleb Rackliffe 
>> wrote:
>>
>> 
>>
>> Alright, the cep-7-sai branch is now merged to trunk!
>>
>> Now we move to addressing the most urgent items from "Phase 2" (
>> CASSANDRA-18473 )
>> before (and in the case of some testing after) the 5.0 freeze...
>>
>> On Wed, Jul 26, 2023 at 6:07 AM Jeremy Hanna 
>> wrote:
>>
>>> Thanks Caleb and Mike and Zhao and Andres and Piotr and everyone else
>>> involved with the SAI implementation!
>>>
>>> On Jul 25, 2023, at 3:01 PM, Caleb Rackliffe 
>>> wrote:
>>>
>>> 
>>> Just a quick update...
>>>
>>> With CASSANDRA-18670
>>>  complete, and
>>> all remaining items in the category of performance optimizations and
>>> further testing, the process of merging to trunk will likely start today,
>>> beginning with a final rebase on the current trunk and J11 and J17 test
>>> runs.
>>>
>>> On Tue, Jul 18, 2023 at 3:47 PM Caleb Rackliffe <
>>> calebrackli...@gmail.com> wrote:
>>>
 Hello there!

 After much toil, the first phase of CEP-7 is nearing completion (see
 CASSANDRA-16052 ).
 There are presently two issues to resolve before we'd like to merge the
 cep-7-sai feature branch and all its goodness to trunk:

 CASSANDRA-18670 
 - Importer should build SSTable indexes successfully before making new
 SSTables readable (in review)

 CASSANDRA-18673 
 - Reduce size of per-SSTable index components (in progress)

 (We've been getting clean CircleCI runs for a while now, and have been
 using the multiplexer to sniff out as much flakiness as possible up front.)

 Once merged to trunk, the next steps are:

 1.) Finish a Harry model that we can use to further fuzz test SAI
 before 5.0 releases (see CASSANDRA-18275
 ). We've done a
 fair amount of fuzz/randomized testing at the component level, but I'd
 still consider Harry (at least around single-partition query use-cases) a
 critical item for us to have confidence before release.

 2.) Start pursuing Phase 2 items as time and our needs allow. (see
 CASSANDRA-18473 
 )

 A reminder, SAI is a secondary index, and therefore is by definition an
 opt-in feature, and has no explicit "feature flag". However, its
 availability to users is still subject to the secondary_indexes_enabled
 guardrail, which currently defaults to allowing creation.

 Any thoughts, questions, or comments on the pre-merge plan here?

>>>

Re: Status Update on CEP-7 Storage Attached Indexes (SAI)

[Jon Haddad]

Very nice!  I'll kick the tires a bit, and add a sai test to tlp-stress 

On 2023/07/26 18:56:29 Caleb Rackliffe wrote:
> Alright, the cep-7-sai branch is now merged to trunk!
> 
> Now we move to addressing the most urgent items from "Phase 2" (
> CASSANDRA-18473 )
> before (and in the case of some testing after) the 5.0 freeze...
> 
> On Wed, Jul 26, 2023 at 6:07 AM Jeremy Hanna 
> wrote:
> 
> > Thanks Caleb and Mike and Zhao and Andres and Piotr and everyone else
> > involved with the SAI implementation!
> >
> > On Jul 25, 2023, at 3:01 PM, Caleb Rackliffe 
> > wrote:
> >
> > 
> > Just a quick update...
> >
> > With CASSANDRA-18670
> >  complete, and all
> > remaining items in the category of performance optimizations and further
> > testing, the process of merging to trunk will likely start today, beginning
> > with a final rebase on the current trunk and J11 and J17 test runs.
> >
> > On Tue, Jul 18, 2023 at 3:47 PM Caleb Rackliffe 
> > wrote:
> >
> >> Hello there!
> >>
> >> After much toil, the first phase of CEP-7 is nearing completion (see
> >> CASSANDRA-16052 ).
> >> There are presently two issues to resolve before we'd like to merge the
> >> cep-7-sai feature branch and all its goodness to trunk:
> >>
> >> CASSANDRA-18670 
> >> - Importer should build SSTable indexes successfully before making new
> >> SSTables readable (in review)
> >>
> >> CASSANDRA-18673 
> >> - Reduce size of per-SSTable index components (in progress)
> >>
> >> (We've been getting clean CircleCI runs for a while now, and have been
> >> using the multiplexer to sniff out as much flakiness as possible up front.)
> >>
> >> Once merged to trunk, the next steps are:
> >>
> >> 1.) Finish a Harry model that we can use to further fuzz test SAI before
> >> 5.0 releases (see CASSANDRA-18275
> >> ). We've done a
> >> fair amount of fuzz/randomized testing at the component level, but I'd
> >> still consider Harry (at least around single-partition query use-cases) a
> >> critical item for us to have confidence before release.
> >>
> >> 2.) Start pursuing Phase 2 items as time and our needs allow. (see
> >> CASSANDRA-18473 )
> >>
> >> A reminder, SAI is a secondary index, and therefore is by definition an
> >> opt-in feature, and has no explicit "feature flag". However, its
> >> availability to users is still subject to the secondary_indexes_enabled
> >> guardrail, which currently defaults to allowing creation.
> >>
> >> Any thoughts, questions, or comments on the pre-merge plan here?
> >>
> >
> 

Re: [ANNOUNCEMENT] Expect failures today. Dropping JDK 8 and adding JDK 11

[Mick Semb Wever]

> FYI,  devbranch (pre-commit) on ci-cassandra.a.o is hardcoded to build on
> jdk8 and jdk11
> so it's not going to work with trunk based patches…
>
> I'll review any patches to fix this, but otherwise focus is
> on CASSANDRA-18594 where a standalone jenkinsfile avoids this problem
> altogether.
>


To clarify this, devbranch was hardcoded to a matrix with jdk8 and jdk11,
with a hack in place to skip jdk11 for <4 branches (at run time).
Dropping jdk8 was not the cause of devbranch no longer working, it's just
not built for it.  These limitations of devbranch (and cassandra-builds)
are being addressed as part of the repeatable ci work.  Note, this does not
affect post-commit ci-cassandra.a.o pipelines (which already is testing
jdk11+17 on trunk).

So… I would like to suggest we abandon CASSANDRA-18701 and we collectively
focus on the intended replacement for this, which is the in-tree scripts
and 18594.  (It might possibly come as part of CASSANDRA-18665 which is an
intermediate ticket).

Stefan and Maxim, I think this impacts the two of you most over the next
few weeks.  In the meantime I suggest using the in-tree scripts as much as
possible, using individual stage jobs under the patches tab in
ci-cassandra.a.o (just ignoring the jdk8 axis failures), and asking others
for final pre-commit ci runs on circleci.  We're close.