Cassandra Java Driver and OpenJDK CRaC

[Radim Vansa]

Hi all,

I would like to make applications using Cassandra Java Driver, 
particularly those built with Spring Boot, Quarkus or similar 
frameworks, work with OpenJDK CRaC project [1]. I've already created a 
patch for Spring Boot [2] but Spring folks think that these changes are 
too dependent on driver internals, suggesting to contribute a support to 
Cassandra directly.


The patch involves closing all connections before checkpoint, and 
re-establishing these after restore. I have implemented that though 
sending a `NodeStateEvent -> FORCED_DOWN` on the bus for all connected 
nodes. As a follow-up I could develop some way to inform the session 
about a new topology e.g. if the cluster addresses change.


Before jumping onto implementing a PR I would like to ask what you think 
is the best approach to do this. I can think of two ways:


1) Native CRaC support

The driver would have a dependency on `org.crac:crac` [3]; this is a 
small (13kB) library that provides the interfaces and a dummy noop 
implementation if the target JVM does not support CRaC. Then 
`DefaultSession` would register a `org.crac.Resource` implementation 
that would handle the checkpoint. This has the advantage of providing 
best fan-out into any project consuming the driver without any further work.


2) Exposing neutral methods

To save frameworks of relying on internals, `DefaultSession` would 
expose `.suspend()` and `.resume()` methods that would implement the 
connection cut-off without importing any dependency. After upgrade to 
latest release, frameworks could use these methods in a way that suits 
them. I wouldn't add those methods to the `CqlSession` interface (as 
that would be breaking change) but only to `DefaultSession`.


Would Cassandra accept either of these, to let people checkpoint 
(snapshot) their applications and restore them within tens of 
milliseconds? Naturally it is possible to close the session object 
completely and create a new one, but the ideal solution would require no 
application changes beyond dependency upgrade.


Btw. I am aware that there is an inherent race between possible topology 
change and shutdown of current nodes (and I am listening for hints that 
would let us prevent that), but it is reasonable to expect that users 
will checkpoint the application in a quiescent state. And if the 
topology update breaks the checkpoint, it is always possible to try it 
again.


Thank you for your opinions and ideas!

Radim Vansa


[1] https://wiki.openjdk.org/display/crac

[2] https://github.com/spring-projects/spring-boot/pull/44505

[3] https://mvnrepository.com/artifact/org.crac/crac/1.5.0

Re: CEP-15 Update

[Jordan West]

Regarding the merge: I have the same question as Jon about rebasing the
feature branch vs a merge to trunk. And again want to recognize that we are
asking more from the maintainers here. I totally acknowledge this is not
zero cost. But I don’t find it terrible that the feature is proven out in
an internal feature branch first if the end result is a better feature for
the community.

Regarding feature flag: I absolutely appreciate that was done and folks
have already stated there should be no impact perf wise when disabled / not
used which I also am glad for. My questions were more about the complexity
introduced to the codebase if the feature did not make it to maturity but
was merged. I’m hearing from Benedict though that it’s rather isolated.
I’ll take a look to form an opinion but if the integration points are super
clear maybe my concern is unwarranted here. I’ll follow up when I’ve had a
chance to look and to Benedict’s point will try to expedite that.

Also to be clear, since it was mentioned in passing, I am in no way saying
-1. I only wanted to bring these questions for discussion to inform my
opinion and because I think it’s a healthy discussion to have ahead of
merging something so large and impactful.

Jordan

On Thu, Mar 6, 2025 at 14:01 Jon Haddad  wrote:

> Hmm... I took a look at the cep-15-accord branch in GitHub, it looks like
> it's several hundred commits behind trunk.  Since you'll need to rebase
> again before merge *anyways*, would it make sense to do it once more, and I
> can publish easy-cass-lab with the latest branch?  If folks have concerns,
> it's easy to fire up a cluster (I do it constantly) and try it out.
>
> I think if we were to do this, out of consideration we should time box the
> amount of time for an evaluation and unless someone raises an objection,
> consider lazy consensus achieved.
>
> Jon
>
>
>
> On Thu, Mar 6, 2025 at 12:46 PM Benedict Elliott Smith <
> bened...@apache.org> wrote:
>
>> Because we want to validate against the latest code in trunk, else we are
>> validating stale behaviours. The cost of rebasing is high, so we do not do
>> it frequently. That means we will likely stop developing OSS-first, as the
>> focus will have to move to our internal branch that satisfies these
>> criteria.
>>
>> Exactly what this might be for upstreaming I cannot say. Personally, I
>> aim to work exclusively on the branch we are stabilising. If that is not
>> trunk, the latency for my contributions being made public might be high, as
>> I have a huge imbalance of over-investment to recoup, and anything
>> unnecessary will be deferred.
>>
>> Since the feature is disabled, and the code is almost entirely isolated,
>> I cannot imagine the cost to the community to removing this work would be
>> very high. But, I do not intend to argue Accord’s case here. I will let you
>> all decide.
>>
>> Please decide soon though, as it shapes our work planning. The positive
>> reception so far had lead me to consider prioritising a move to trunk-first
>> development within the next week or two, and the associated work that
>> entails. However, if that was optimistic we will have to shift our plans.
>>
>>
>>
>> On 6 Mar 2025, at 20:16, Jordan West  wrote:
>>
>> The work and effort in accord has been amazing. And I’m sure it sets a
>> new standard for code quality and correctness testing which I’m also
>> entirely behind. I also trust the folks working on it want to take it to
>> the a fully production ready solution. But I’m worried about circumstances
>> out of our control leaving us with a very complex feature that isn’t
>> complete.
>>
>> I do have some questions. Could folks help me better understand why
>> testing real workloads necessitates a merge (my understanding from the
>> original reason is this is the impetus for why we would merge now)? Also I
>> think the performance and scheme change caveats are rather large ones. One
>> of accords promise was better performance and I think making schema changes
>> with nodes down not being supported is a big gap. Could we have some
>> criteria like “supports all the operations PaxosV2 supports” or “performs
>> as well or better than PaxosV2 on [workload(s)]”?
>>
>> I understand waiting asks a lot of the authors in terms of baring the
>> burden of a more complex merge. But I think we also need to consider what
>> merging is asking the community to bear if the worst happens and we are
>> unable to take the feature from its current state to something that can be
>> widely used in production.
>>
>>
>> Jordan
>>
>>
>> On Wed, Mar 5, 2025 at 15:52 Blake Eggleston 
>> wrote:
>>
>>> +1 to merging it
>>>
>>> On Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:
>>>
>>> You have my +1
>>>
>>> On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:
>>> >
>>> > Correct, these caveats should only apply to tables that have opted-in
>>> to accord.
>>> >
>>> > On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:
>>> >
>>> > 
>>> > So great to see all this hard work a

Re: CEP-15 Update

[Jeff Jirsa]

Having played with the branch and confirmed it’s relatively isolated, I would love to see it mergedOn Mar 6, 2025, at 8:44 PM, Benedict Elliott Smith  wrote:Because we want to validate against the latest code in trunk, else we are validating stale behaviours. The cost of rebasing is high, so we do not do it frequently. That means we will likely stop developing OSS-first, as the focus will have to move to our internal branch that satisfies these criteria.Exactly what this might be for upstreaming I cannot say. Personally, I aim to work exclusively on the branch we are stabilising. If that is not trunk, the latency for my contributions being made public might be high, as I have a huge imbalance of over-investment to recoup, and anything unnecessary will be deferred.Since the feature is disabled, and the code is almost entirely isolated, I cannot imagine the cost to the community to removing this work would be very high. But, I do not intend to argue Accord’s case here. I will let you all decide.Please decide soon though, as it shapes our work planning. The positive reception so far had lead me to consider prioritising a move to trunk-first development within the next week or two, and the associated work that entails. However, if that was optimistic we will have to shift our plans.On 6 Mar 2025, at 20:16, Jordan West  wrote:The work and effort in accord has been amazing. And I’m sure it sets a new standard for code quality and correctness testing which I’m also entirely behind. I also trust the folks working on it want to take it to the a fully production ready solution. But I’m worried about circumstances out of our control leaving us with a very complex feature that isn’t complete. I do have some questions. Could folks help me better understand why testing real workloads necessitates a merge (my understanding from the original reason is this is the impetus for why we would merge now)? Also I think the performance and scheme change caveats are rather large ones. One of accords promise was better performance and I think making schema changes with nodes down not being supported is a big gap. Could we have some criteria like “supports all the operations PaxosV2 supports” or “performs as well or better than PaxosV2 on [workload(s)]”? I understand waiting asks a lot of the authors in terms of baring the burden of a more complex merge. But I think we also need to consider what merging is asking the community to bear if the worst happens and we are unable to take the feature from its current state to something that can be widely used in production.Jordan On Wed, Mar 5, 2025 at 15:52 Blake Eggleston  wrote:+1 to merging itOn Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:You have my +1On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:>> Correct, these caveats should only apply to tables that have opted-in to accord.>> On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:>> > So great to see all this hard work about to pay off!>> On the questions/concerns front, the only concern I would have towards merging this to trunk is if any of the caveats apply when someone is not using Accord.  Assuming they only apply when the feature flag is enabled, I see no reason not to get this merged into trunk once everyone involved is happy with the state of it.>> -Jeremiah>> On Mar 5, 2025 at 12:15:23 PM, Benedict Elliott Smith  wrote: That depends on all of you lovely people :D I think we should have finished merging everything we want before QA by ~Monday; certainly not much later. I think we have some upgrade and python dtest failures to address as well. So it could be pretty soon if the community is supportive. On 5 Mar 2025, at 17:22, Patrick McFadin  wrote:>> What is the timing for starting the merge process? I'm asking because I have (yet another) presentation and this would be a cool update.>> On Wed, Mar 5, 2025 at 1:22 AM Benedict Elliott Smith  wrote: > > Thanks everyone. > > Jon - your help will be greatly appreciated. We’ll let you know when we’ve got the cycles to invest in performance work (hopefully fairly soon). I expect the first step will be improving visibility so we can better understand what the system is doing (particularly the caching layers), but we can dig in together when ready. > > On 4 Mar 2025, at 18:15, Jon Haddad  wrote: > > Very exciting! > > I have a client that's very interested in Accord, so I should have budget to dig into it, especially on the performance side of things. > > Jon > > On Tue, Mar 4, 2025 at 9:57 AM Dmitry Konstantinov  wrote: >> >> Thank you to all Accord and TCM contributors, it is really exciting to see a development of such huge and wonderful features moving forward and opening the door to the new Cassandra ep

Re: stalled PR #3585 (CASSANDRA-14098: integer overflow fix)

[Danny Faught]

I have assigned the JIRA ticket to myself and added a comment. Does the
status need to be updated?

Let me know if we should have issued the PR against a release branch
instead of trunk.

-Danny

On Tue, Jan 21, 2025 at 1:41 PM Dinesh Joshi  wrote:

> Hi Danny,
>
> Thanks for pinging this ist. It would be helpful for you to indicate on
> that JIRA that you're picking up the issue. Right now the JIRA is not
> assigned to anybody. I don't think you have a JIRA account. Could you
> please request[1] one? Once you have it, I will assign you the ticket and
> you can post your update in the ticket with the git branch with your fix.
> Please respond back to the list and hopefully someone will review your
> ticket and merge it in.
>
> It would be great for you to review the general guidelines[2] for
> contributing to the project.
>
> thanks,
>
> Dinesh
>
>
> [1] https://selfserve.apache.org/jira-account.html
> [2] https://cassandra.apache.org/_/development/patches.html
>
> On Mon, Jan 20, 2025 at 1:50 PM Danny Faught 
> wrote:
>
>> Hi! New contributor to the project here. I opened PR 3585
>> (CASSANDRA-14098: integer overflow fix)
>>  last September to
>> address a Jira ticket that has been open for quite some time (
>> https://issues.apache.org/jira/browse/CASSANDRA-14098).
>>
>> I can imagine it's a low priority for the project, and if it's simply
>> waiting for someone to have the time to review it, no worries. But if I
>> missed something in the process that I should have done, please let me
>> know.
>>
>> I'm considering using this PR as a code sample for a conference
>> presentation, and if there is feedback on whether any changes are needed,
>> that would give me the opportunity to improve the code first.
>>
>> -Danny
>>
>>

Re: CEP-15 Update

[Benedict]

There are essentially three possible timelines to choose from here: 1) We agree in the next few days to merge to trunk. We will then prioritise rebasing onto trunk and resolving any pre-merge items starting next week.2) There’s some more debate and agreement to merge to trunk in a week or two. In the meantime we will shift to internal-first development but we’ll likely prioritise the above work as soon as we can, which may be in a few weeks, so we can shift to trunk first development.3) We don’t agree to merge accord anytime soon, so we shift to internal-first development for the time being. I’m not sure when we will prioritise any of the above.Our resources are finite and we’ve exhausted them (literally), so it’s pretty much pick one of the above. I don’t really mind which you pick, but I won’t personally be prioritising merge after this third attempt.On 6 Mar 2025, at 22:01, Jon Haddad  wrote:Hmm... I took a look at the cep-15-accord branch in GitHub, it looks like it's several hundred commits behind trunk.  Since you'll need to rebase again before merge *anyways*, would it make sense to do it once more, and I can publish easy-cass-lab with the latest branch?  If folks have concerns, it's easy to fire up a cluster (I do it constantly) and try it out.I think if we were to do this, out of consideration we should time box the amount of time for an evaluation and unless someone raises an objection, consider lazy consensus achieved.JonOn Thu, Mar 6, 2025 at 12:46 PM Benedict Elliott Smith  wrote:Because we want to validate against the latest code in trunk, else we are validating stale behaviours. The cost of rebasing is high, so we do not do it frequently. That means we will likely stop developing OSS-first, as the focus will have to move to our internal branch that satisfies these criteria.Exactly what this might be for upstreaming I cannot say. Personally, I aim to work exclusively on the branch we are stabilising. If that is not trunk, the latency for my contributions being made public might be high, as I have a huge imbalance of over-investment to recoup, and anything unnecessary will be deferred.Since the feature is disabled, and the code is almost entirely isolated, I cannot imagine the cost to the community to removing this work would be very high. But, I do not intend to argue Accord’s case here. I will let you all decide.Please decide soon though, as it shapes our work planning. The positive reception so far had lead me to consider prioritising a move to trunk-first development within the next week or two, and the associated work that entails. However, if that was optimistic we will have to shift our plans.On 6 Mar 2025, at 20:16, Jordan West  wrote:The work and effort in accord has been amazing. And I’m sure it sets a new standard for code quality and correctness testing which I’m also entirely behind. I also trust the folks working on it want to take it to the a fully production ready solution. But I’m worried about circumstances out of our control leaving us with a very complex feature that isn’t complete. I do have some questions. Could folks help me better understand why testing real workloads necessitates a merge (my understanding from the original reason is this is the impetus for why we would merge now)? Also I think the performance and scheme change caveats are rather large ones. One of accords promise was better performance and I think making schema changes with nodes down not being supported is a big gap. Could we have some criteria like “supports all the operations PaxosV2 supports” or “performs as well or better than PaxosV2 on [workload(s)]”? I understand waiting asks a lot of the authors in terms of baring the burden of a more complex merge. But I think we also need to consider what merging is asking the community to bear if the worst happens and we are unable to take the feature from its current state to something that can be widely used in production.Jordan On Wed, Mar 5, 2025 at 15:52 Blake Eggleston  wrote:+1 to merging itOn Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:You have my +1On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:>> Correct, these caveats should only apply to tables that have opted-in to accord.>> On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:>> > So great to see all this hard work about to pay off!>> On the questions/concerns front, the only concern I would have towards merging this to trunk is if any of the caveats apply when someone is not using Accord.  Assuming they only apply when the feature flag is enabled, I see no reason not to get this merged into trunk once everyone involved is happy with the state of it.>> -Jeremiah>> On Mar 5, 2025 at 12:15:23 PM, Benedict Elliott Smith  wrote: That depends on all of you lovely people :D I think we should have finished merging everything we want before QA by ~Monday; certainly

Re: CEP-15 Update

[Jordan West]

A couple questions:
- are there some graphs or raw data that has or could be shared to help
quantify what is meant by “SLOW”? If not maybe worth taking Jon up on his
offer of an ECL build after a last rebase and a short time box? I could
likely take it for a spin over the weekend. Might need a couple tweaks to
ECS but they would be worth it anyways.

- is there a known path forward to fix the drop schema w nodes down issue
and anything written on it?

Jordan

On Thu, Mar 6, 2025 at 17:08 Blake Eggleston  wrote:

> > I don’t find it terrible that the feature is proven out in an internal
> feature branch first if the end result is a better feature for the
> community.
>
> I don’t think it’s going to be merged and designated as production ready.
> I think the intention is to mark it experimental/alpha/beta/whatever while
> kinks are ironed out. This is a good way to approach development of large
> features like this.
>
> At this point, keeping accord in a feature branch isn’t going to result in
> a better feature for the community. The risk of abandonment is vanishingly
> small, and the costs of keeping it unmerged are increasing. It doesn’t just
> create developer friction, but it also keeps the feature out of the hands
> of people who want to play around with it that will be a source of feedback
> (bugs, perf, ui, etc) for the authors.
>
> I do think it’s healthy to talk this out though.
>
> On Thu, Mar 6, 2025, at 4:41 PM, Jeff Jirsa wrote:
>
>
> Having played with the branch and confirmed it’s relatively isolated, I
> would love to see it merged
>
>
> On Mar 6, 2025, at 8:44 PM, Benedict Elliott Smith 
> wrote:
>
> Because we want to validate against the latest code in trunk, else we are
> validating stale behaviours. The cost of rebasing is high, so we do not do
> it frequently. That means we will likely stop developing OSS-first, as the
> focus will have to move to our internal branch that satisfies these
> criteria.
>
> Exactly what this might be for upstreaming I cannot say. Personally, I aim
> to work exclusively on the branch we are stabilising. If that is not trunk,
> the latency for my contributions being made public might be high, as I have
> a huge imbalance of over-investment to recoup, and anything unnecessary
> will be deferred.
>
> Since the feature is disabled, and the code is almost entirely isolated, I
> cannot imagine the cost to the community to removing this work would be
> very high. But, I do not intend to argue Accord’s case here. I will let you
> all decide.
>
> Please decide soon though, as it shapes our work planning. The positive
> reception so far had lead me to consider prioritising a move to trunk-first
> development within the next week or two, and the associated work that
> entails. However, if that was optimistic we will have to shift our plans.
>
>
>
> On 6 Mar 2025, at 20:16, Jordan West  wrote:
>
> The work and effort in accord has been amazing. And I’m sure it sets a new
> standard for code quality and correctness testing which I’m also entirely
> behind. I also trust the folks working on it want to take it to the a fully
> production ready solution. But I’m worried about circumstances out of our
> control leaving us with a very complex feature that isn’t complete.
>
> I do have some questions. Could folks help me better understand why
> testing real workloads necessitates a merge (my understanding from the
> original reason is this is the impetus for why we would merge now)? Also I
> think the performance and scheme change caveats are rather large ones. One
> of accords promise was better performance and I think making schema changes
> with nodes down not being supported is a big gap. Could we have some
> criteria like “supports all the operations PaxosV2 supports” or “performs
> as well or better than PaxosV2 on [workload(s)]”?
>
> I understand waiting asks a lot of the authors in terms of baring the
> burden of a more complex merge. But I think we also need to consider what
> merging is asking the community to bear if the worst happens and we are
> unable to take the feature from its current state to something that can be
> widely used in production.
>
> Jordan
>
>
> On Wed, Mar 5, 2025 at 15:52 Blake Eggleston  wrote:
>
>
> +1 to merging it
>
> On Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:
>
> You have my +1
>
> On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:
> >
> > Correct, these caveats should only apply to tables that have opted-in to
> accord.
> >
> > On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:
> >
> > 
> > So great to see all this hard work about to pay off!
> >
> > On the questions/concerns front, the only concern I would have towards
> merging this to trunk is if any of the caveats apply when someone is not
> using Accord.  Assuming they only apply when the feature flag is enabled, I
> see no reason not to get this merged into trunk once everyone involved is
> happy with the state of it.
> >
> > -Jeremiah
> >
> > On Mar 5, 2025 at 12:15:2

Re: CEP-15 Update

[Blake Eggleston]

> I don’t find it terrible that the feature is proven out in an internal 
> feature branch first if the end result is a better feature for the community. 

I don’t think it’s going to be merged and designated as production ready. I 
think the intention is to mark it experimental/alpha/beta/whatever while kinks 
are ironed out. This is a good way to approach development of large features 
like this.

At this point, keeping accord in a feature branch isn’t going to result in a 
better feature for the community. The risk of abandonment is vanishingly small, 
and the costs of keeping it unmerged are increasing. It doesn’t just create 
developer friction, but it also keeps the feature out of the hands of people 
who want to play around with it that will be a source of feedback (bugs, perf, 
ui, etc) for the authors.

I do think it’s healthy to talk this out though.

On Thu, Mar 6, 2025, at 4:41 PM, Jeff Jirsa wrote:
> 
> Having played with the branch and confirmed it’s relatively isolated, I would 
> love to see it merged
> 
> 
>> On Mar 6, 2025, at 8:44 PM, Benedict Elliott Smith  
>> wrote:
>> Because we want to validate against the latest code in trunk, else we are 
>> validating stale behaviours. The cost of rebasing is high, so we do not do 
>> it frequently. That means we will likely stop developing OSS-first, as the 
>> focus will have to move to our internal branch that satisfies these criteria.
>> 
>> Exactly what this might be for upstreaming I cannot say. Personally, I aim 
>> to work exclusively on the branch we are stabilising. If that is not trunk, 
>> the latency for my contributions being made public might be high, as I have 
>> a huge imbalance of over-investment to recoup, and anything unnecessary will 
>> be deferred.
>> 
>> Since the feature is disabled, and the code is almost entirely isolated, I 
>> cannot imagine the cost to the community to removing this work would be very 
>> high. But, I do not intend to argue Accord’s case here. I will let you all 
>> decide.
>> 
>> Please decide soon though, as it shapes our work planning. The positive 
>> reception so far had lead me to consider prioritising a move to trunk-first 
>> development within the next week or two, and the associated work that 
>> entails. However, if that was optimistic we will have to shift our plans.
>> 
>> 
>> 
>>> On 6 Mar 2025, at 20:16, Jordan West  wrote:
>>> 
>>> The work and effort in accord has been amazing. And I’m sure it sets a new 
>>> standard for code quality and correctness testing which I’m also entirely 
>>> behind. I also trust the folks working on it want to take it to the a fully 
>>> production ready solution. But I’m worried about circumstances out of our 
>>> control leaving us with a very complex feature that isn’t complete. 
>>> 
>>> I do have some questions. Could folks help me better understand why testing 
>>> real workloads necessitates a merge (my understanding from the original 
>>> reason is this is the impetus for why we would merge now)? Also I think the 
>>> performance and scheme change caveats are rather large ones. One of accords 
>>> promise was better performance and I think making schema changes with nodes 
>>> down not being supported is a big gap. Could we have some criteria like 
>>> “supports all the operations PaxosV2 supports” or “performs as well or 
>>> better than PaxosV2 on [workload(s)]”? 
>>> 
>>> I understand waiting asks a lot of the authors in terms of baring the 
>>> burden of a more complex merge. But I think we also need to consider what 
>>> merging is asking the community to bear if the worst happens and we are 
>>> unable to take the feature from its current state to something that can be 
>>> widely used in production.
>>> 
>>> Jordan 
>>> 
>>> 
>>> On Wed, Mar 5, 2025 at 15:52 Blake Eggleston  wrote:
 __
 +1 to merging it
 
 On Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:
> You have my +1
> 
> On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:
> >
> > Correct, these caveats should only apply to tables that have opted-in 
> > to accord.
> >
> > On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:
> >
> > 
> > So great to see all this hard work about to pay off!
> >
> > On the questions/concerns front, the only concern I would have towards 
> > merging this to trunk is if any of the caveats apply when someone is 
> > not using Accord.  Assuming they only apply when the feature flag is 
> > enabled, I see no reason not to get this merged into trunk once 
> > everyone involved is happy with the state of it.
> >
> > -Jeremiah
> >
> > On Mar 5, 2025 at 12:15:23 PM, Benedict Elliott Smith 
> >  wrote:
> >>
> >> That depends on all of you lovely people :D
> >>
> >> I think we should have finished merging everything we want before QA 
> >> by ~Monday; certainly not much later.
> >>
> >> I think we have some upgrade and python dtest 

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Jon Haddad]

Assuming everything else is identical, might not matter for S3. However,
not every object store has a filesystem mount.

Regarding sprawling dependencies, we can always make the provider specific
libraries available as a separate download and put them on their own thread
with a separate class path. I think in JVM dtest does this already.
Someone just started asking about IAM for login, it sounds like a similar
problem.


On Thu, Mar 6, 2025 at 12:53 AM Benedict  wrote:

> I think another way of saying what Stefan may be getting at is what does a
> library give us that an appropriately configured mount dir doesn’t?
>
> We don’t want to treat S3 the same as local disk, but this can be achieved
> easily with config. Is there some other benefit of direct integration? Well
> defined exceptions if we need to distinguish cases is one that maybe
> springs to mind but perhaps there are others?
>
>
> On 6 Mar 2025, at 08:39, Štefan Miklošovič  wrote:
>
> 
>
> That is cool but this still does not show / explain how it would look like
> when it comes to dependencies needed for actually talking to storages like
> s3.
>
> Maybe I am missing something here and please explain when I am mistaken
> but If I understand that correctly, for talking to s3 we would need to use
> a library like this, right? (1). So that would be added among Cassandra
> dependencies? Hence Cassandra starts to be biased against s3? Why s3? Every
> time somebody comes up with a new remote storage support, that would be
> added to classpath as well? How are these dependencies going to play with
> each other and with Cassandra in general? Will all these storage
> provider libraries for arbitrary clouds be even compatible with Cassandra
> licence-wise?
>
> I am sorry I keep repeating these questions but this part of that I just
> don't get at all.
>
> We can indeed add an API for this, sure sure, why not. But for people who
> do not want to deal with this at all and just be OK with a FS mounted, why
> would we block them doing that?
>
> (1)
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/pom.xml
>
> On Wed, Mar 5, 2025 at 3:28 PM Mick Semb Wever  wrote:
>
>>.
>>
>>
>> It’s not an area where I can currently dedicate engineering effort. But
>>> if others are interested in contributing a feature like this, I’d see it as
>>> valuable for the project and would be happy to collaborate on
>>> design/architecture/goals.
>>>
>>
>>
>> Jake mentioned 17 months ago a custom FileSystemProvider we could offer.
>>
>> None of us at DataStax has gotten around to providing that, but to
>> quickly throw something over the wall this is it:
>>
>> https://github.com/datastax/cassandra/blob/main/src/java/org/apache/cassandra/io/storage/StorageProvider.java
>>
>>   (with a few friend classes under o.a.c.io.util)
>>
>> We then have a RemoteStorageProvider, private in another repo, that
>> implements that and also provides the RemoteFileSystemProvider that Jake
>> refers to.
>>
>> Hopefully that's a start to get people thinking about CEP level details,
>> while we get a cleaned abstract of RemoteStorageProvider and friends to
>> offer.
>>
>>

Re: PHP Client Driver

[Brandon Williams]

Hi Michael,

I created https://issues.apache.org/jira/browse/CASSANDRA-20406 to get
this changed.

Kind Regards,
Brandon

On Tue, Mar 4, 2025 at 5:34 AM Michael Roosz  wrote:
>
> Hello Cassandra developers,
>
> since the official Cassandra php client driver list currently only contains 
> outdated and unmaintained projects, could you please consider adding my php 
> client driver to this list?
> https://cassandra.apache.org/doc/latest/cassandra/getting-started/drivers.html#php
> https://github.com/MichaelRoosz/php-cassandra
>
> Best Regards,
> Michael

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Benedict]

I think another way of saying what Stefan may be getting at is: what does a library give us that an appropriately configured mount dir doesn’t?We don’t want to treat S3 the same as local disk, but this can be achieved easily with config. Is there some other benefit of direct integration? Well defined exceptions if we need to distinguish cases is one that maybe springs to mind but perhaps there are others?On 6 Mar 2025, at 08:39, Štefan Miklošovič  wrote:That is cool but this still does not show / explain how it would look like when it comes to dependencies needed for actually talking to storages like s3. Maybe I am missing something here and please explain when I am mistaken but If I understand that correctly, for talking to s3 we would need to use a library like this, right? (1). So that would be added among Cassandra dependencies? Hence Cassandra starts to be biased against s3? Why s3? Every time somebody comes up with a new remote storage support, that would be added to classpath as well? How are these dependencies going to play with each other and with Cassandra in general? Will all these storage provider libraries for arbitrary clouds be even compatible with Cassandra licence-wise?I am sorry I keep repeating these questions but this part of that I just don't get at all. We can indeed add an API for this, sure sure, why not. But for people who do not want to deal with this at all and just be OK with a FS mounted, why would we block them doing that?(1) https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/pom.xmlOn Wed, Mar 5, 2025 at 3:28 PM Mick Semb Wever  wrote:   .  It’s not an area where I can currently dedicate engineering effort. But if others are interested in contributing a feature like this, I’d see it as valuable for the project and would be happy to collaborate on design/architecture/goals.Jake mentioned 17 months ago a custom FileSystemProvider we could offer.None of us at DataStax has gotten around to providing that, but to quickly throw something over the wall this is it: https://github.com/datastax/cassandra/blob/main/src/java/org/apache/cassandra/io/storage/StorageProvider.java   (with a few friend classes under o.a.c.io.util)We then have a RemoteStorageProvider, private in another repo, that implements that and also provides the RemoteFileSystemProvider that Jake refers to.Hopefully that's a start to get people thinking about CEP level details, while we get a cleaned abstract of RemoteStorageProvider and friends to offer.

Re: CEP-15 Update

[Jordan West]

The work and effort in accord has been amazing. And I’m sure it sets a new
standard for code quality and correctness testing which I’m also entirely
behind. I also trust the folks working on it want to take it to the a fully
production ready solution. But I’m worried about circumstances out of our
control leaving us with a very complex feature that isn’t complete.

I do have some questions. Could folks help me better understand why testing
real workloads necessitates a merge (my understanding from the original
reason is this is the impetus for why we would merge now)? Also I think the
performance and scheme change caveats are rather large ones. One of accords
promise was better performance and I think making schema changes with nodes
down not being supported is a big gap. Could we have some criteria like
“supports all the operations PaxosV2 supports” or “performs as well or
better than PaxosV2 on [workload(s)]”?

I understand waiting asks a lot of the authors in terms of baring the
burden of a more complex merge. But I think we also need to consider what
merging is asking the community to bear if the worst happens and we are
unable to take the feature from its current state to something that can be
widely used in production.


Jordan


On Wed, Mar 5, 2025 at 15:52 Blake Eggleston  wrote:

> +1 to merging it
>
> On Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:
>
> You have my +1
>
> On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:
> >
> > Correct, these caveats should only apply to tables that have opted-in to
> accord.
> >
> > On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:
> >
> > 
> > So great to see all this hard work about to pay off!
> >
> > On the questions/concerns front, the only concern I would have towards
> merging this to trunk is if any of the caveats apply when someone is not
> using Accord.  Assuming they only apply when the feature flag is enabled, I
> see no reason not to get this merged into trunk once everyone involved is
> happy with the state of it.
> >
> > -Jeremiah
> >
> > On Mar 5, 2025 at 12:15:23 PM, Benedict Elliott Smith <
> bened...@apache.org> wrote:
> >>
> >> That depends on all of you lovely people :D
> >>
> >> I think we should have finished merging everything we want before QA by
> ~Monday; certainly not much later.
> >>
> >> I think we have some upgrade and python dtest failures to address as
> well.
> >>
> >> So it could be pretty soon if the community is supportive.
> >>
> >> On 5 Mar 2025, at 17:22, Patrick McFadin  wrote:
> >>
> >>
> >> What is the timing for starting the merge process? I'm asking because
> >>
> >> I have (yet another) presentation and this would be a cool update.
> >>
> >>
> >> On Wed, Mar 5, 2025 at 1:22 AM Benedict Elliott Smith
> >>
> >>  wrote:
> >>
> >> >
> >>
> >> > Thanks everyone.
> >>
> >> >
> >>
> >> > Jon - your help will be greatly appreciated. We’ll let you know when
> we’ve got the cycles to invest in performance work (hopefully fairly soon).
> I expect the first step will be improving visibility so we can better
> understand what the system is doing (particularly the caching layers), but
> we can dig in together when ready.
> >>
> >> >
> >>
> >> > On 4 Mar 2025, at 18:15, Jon Haddad  wrote:
> >>
> >> >
> >>
> >> > Very exciting!
> >>
> >> >
> >>
> >> > I have a client that's very interested in Accord, so I should have
> budget to dig into it, especially on the performance side of things.
> >>
> >> >
> >>
> >> > Jon
> >>
> >> >
> >>
> >> > On Tue, Mar 4, 2025 at 9:57 AM Dmitry Konstantinov <
> netud...@gmail.com> wrote:
> >>
> >> >>
> >>
> >> >> Thank you to all Accord and TCM contributors, it is really exciting
> to see a development of such huge and wonderful features moving forward and
> opening the door to the new Cassandra epoch!
> >>
> >> >>
> >>
> >> >> On Tue, 4 Mar 2025 at 20:45, Blake Eggleston 
> wrote:
> >>
> >> >>>
> >>
> >> >>> Thanks Benedict!
> >>
> >> >>>
> >>
> >> >>> I’m really excited to see accord reach this milestone, even with
> these caveats. You seem to have left yourself off the list of contributors
> though, even though you’ve been a central figure in its development :) So
> thanks to all accord & tcm contributors, including Benedict, for making
> this possible!
> >>
> >> >>>
> >>
> >> >>> On Tue, Mar 4, 2025, at 8:00 AM, Benedict Elliott Smith wrote:
> >>
> >> >>>
> >>
> >> >>> Hi everyone,
> >>
> >> >>>
> >>
> >> >>> It’s been exactly 3.5 years since the first commit to
> cassandra-accord. Yes, really, it’s been that long.
> >>
> >> >>>
> >>
> >> >>> We will be starting to validate the feature against real workloads
> in the near future, so we can’t sensibly push off merging much longer. The
> following is a brief run-down of the state of play. There are no known
> bugs, but there remain a number of caveats we will be incrementally
> addressing in the run-up to a full release:
> >>
> >> >>>
> >>
> >> >>> [1] Accord is likely to be SLOW until further optimisations are
> implemented
> >>
> >> >>> [2]

Re: [DISCUSS] Plugins and dependencies

[Jon Haddad]

This has been on my mind for a while, and I think it's a great idea.
Someone shouldn't need all the C* internals just to implement an interface.

On Thu, Mar 6, 2025 at 1:08 PM Joel Shepherd  wrote:

> Splitting this out from the CEP-36 thread.
>
> I agree: dependency collisions at run-time are a problem. It's made even
> worse by the possibility of users using multiple plugins (authn, authz,
> compression, storage, etc.).
>
> It also cuts two ways. E.g. the interfaces that plugin authenticators need
> to implement are defined in org.apache.cassandra.auth, so as far as I know
> the plugin has to take a build-time dependency on the main Cassandra module
> itself, and pull in all of its dependencies. (I'd love to be told that I'm
> mistaken.) In addition to the risk of version conflicts, it increases the
> risk of a change to Cassandra's own dependencies inadvertently breaking a
> plugin that's taken a transitive dependency. Might be bad form on the
> plugin's part, but certainly possible.
>
> I've gotten the impression that there's not a lot of enthusiasm for
> breaking apart the main Cassandra module, but I have wondered if it'd be
> worth making an exception for the interfaces plugins are supposed to code
> against. It'd be nice to depend on those without pulling in the rest of the
> project, and it'd be another step towards reducing the risk of plugins
> breaking because of dependency changes in the main project.
>
> -- Joel.
> On 3/6/2025 10:52 AM, Jon Haddad wrote:
>
> Hey Joel, thanks for chiming in!
>
> Regarding dependencies - while it's possible to provide pluggable
> interfaces, the issue I'm concerned about is conflicting versions of
> transitive dependencies at runtime.  For example, I used a java agent that
> had a different version of snakeyaml, and it ended up breaking C*'s startup
> sequence [1].  I suggest putting external modules on separate threads with
> their own classpath to avoid this issue.
>
> I think there's quite a bit of overlap between the two desires expressed
> in this thread, even though they achieve very different results.  I
> personally can't see myself using something that treats an object store as
> cold storage where SSTables are moved (implying they weren't there before),
> and I've expressed my concerns with this, but other folks seem to want it
> and that's OK.  I feel very strongly that treating local storage as a cache
> with the full dataset on object store is a better approach, but ultimately
> different people have different priorities.  Either way, stuff is moved to
> object store at some point, and pulled to the local disk on demand.
>
> I am *firmly* of the position that this CEP should not exclude the local
> storage as cache option, and should be accounted for in the design.
>
> Jon
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19663
>
>
> On Thu, Mar 6, 2025 at 10:31 AM Joel Shepherd  wrote:
>
>> On 3/6/2025 7:16 AM, Jon Haddad wrote:
>>
>> Assuming everything else is identical, might not matter for S3. However,
>> not every object store has a filesystem mount.
>>
>> Regarding sprawling dependencies, we can always make the provider
>> specific libraries available as a separate download and put them on their
>> own thread with a separate class path. I think in JVM dtest does this
>> already.  Someone just started asking about IAM for login, it sounds like a
>> similar problem.
>>
>> That was me. :-) Cassandra's auth already has fairly well defined
>> interfaces and a plug-in mechanism, so it's easy to vend alternative auth
>> solutions without polluting the main project's dependency graph, at
>> build-time anyway. A similar approach could be beneficial for CEP-36,
>> particularly (IMO) for cold-storage purposes. I suspect decoupling
>> pluggable alternate channel proxies for cold storage from configurable
>> alternate channel proxies for redirecting data locally to free up space,
>> migrate to a different storage device, etc., would make both easier. The
>> CEP seems to be trying to do both, but they smell like pretty different
>> goals to me.
>>
>> Thanks -- Joel.
>>
>>
>> On Thu, Mar 6, 2025 at 12:53 AM Benedict  wrote:
>>
>>> I think another way of saying what Stefan may be getting at is what does
>>> a library give us that an appropriately configured mount dir doesn’t?
>>>
>>> We don’t want to treat S3 the same as local disk, but this can be
>>> achieved easily with config. Is there some other benefit of direct
>>> integration? Well defined exceptions if we need to distinguish cases is one
>>> that maybe springs to mind but perhaps there are others?
>>>
>>>
>>> On 6 Mar 2025, at 08:39, Štefan Miklošovič 
>>> wrote:
>>>
>>> 
>>>
>>> That is cool but this still does not show / explain how it would look
>>> like when it comes to dependencies needed for actually talking to storages
>>> like s3.
>>>
>>> Maybe I am missing something here and please explain when I am mistaken
>>> but If I understand that correctly, for talking to s3 we would need to

Re: [DISCUSS] Plugins and dependencies

[Josh McKenzie]

> I've gotten the impression that there's not a lot of enthusiasm for breaking 
> apart the main Cassandra module, but I have wondered if it'd be worth making 
> an exception for the interfaces plugins are supposed to code against
Oh, there's *plenty* of enthusiasm. There's been a shortage of consensus 
however. *For now. *:D

I think breaking out the interfaces first makes a lot of sense as that'd allow 
us to focus almost purely on build dependency and environmental factors w/out 
having to reason through implementation code movements and encapsulation 
breakage. I believe there's folks working on exploring the current build system 
through the lens of requirements to break out shared deps; I'll see if I can't 
rustle them up.

On Thu, Mar 6, 2025, at 4:06 PM, Joel Shepherd wrote:
> Splitting this out from the CEP-36 thread.
> 
> I agree: dependency collisions at run-time are a problem. It's made even 
> worse by the possibility of users using multiple plugins (authn, authz, 
> compression, storage, etc.).
> 
> It also cuts two ways. E.g. the interfaces that plugin authenticators need to 
> implement are defined in org.apache.cassandra.auth, so as far as I know the 
> plugin has to take a build-time dependency on the main Cassandra module 
> itself, and pull in all of its dependencies. (I'd love to be told that I'm 
> mistaken.) In addition to the risk of version conflicts, it increases the 
> risk of a change to Cassandra's own dependencies inadvertently breaking a 
> plugin that's taken a transitive dependency. Might be bad form on the 
> plugin's part, but certainly possible.
> 
> I've gotten the impression that there's not a lot of enthusiasm for breaking 
> apart the main Cassandra module, but I have wondered if it'd be worth making 
> an exception for the interfaces plugins are supposed to code against. It'd be 
> nice to depend on those without pulling in the rest of the project, and it'd 
> be another step towards reducing the risk of plugins breaking because of 
> dependency changes in the main project.
> 
> -- Joel.
> 
> On 3/6/2025 10:52 AM, Jon Haddad wrote:
>> Hey Joel, thanks for chiming in!
>> 
>> Regarding dependencies - while it's possible to provide pluggable 
>> interfaces, the issue I'm concerned about is conflicting versions of 
>> transitive dependencies at runtime.  For example, I used a java agent that 
>> had a different version of snakeyaml, and it ended up breaking C*'s startup 
>> sequence [1].  I suggest putting external modules on separate threads with 
>> their own classpath to avoid this issue. 
>> 
>> I think there's quite a bit of overlap between the two desires expressed in 
>> this thread, even though they achieve very different results.  I personally 
>> can't see myself using something that treats an object store as cold storage 
>> where SSTables are moved (implying they weren't there before), and I've 
>> expressed my concerns with this, but other folks seem to want it and that's 
>> OK.  I feel very strongly that treating local storage as a cache with the 
>> full dataset on object store is a better approach, but ultimately different 
>> people have different priorities.  Either way, stuff is moved to object 
>> store at some point, and pulled to the local disk on demand. 
>> 
>> I am *firmly* of the position that this CEP should not exclude the local 
>> storage as cache option, and should be accounted for in the design.
>> 
>> Jon
>> 
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19663
>> 
>> 
>> On Thu, Mar 6, 2025 at 10:31 AM Joel Shepherd  wrote:
>>> On 3/6/2025 7:16 AM, Jon Haddad wrote:
 Assuming everything else is identical, might not matter for S3. However, 
 not every object store has a filesystem mount. 
 
 Regarding sprawling dependencies, we can always make the provider specific 
 libraries available as a separate download and put them on their own 
 thread with a separate class path. I think in JVM dtest does this already. 
  Someone just started asking about IAM for login, it sounds like a similar 
 problem.
>>> That was me. :-) Cassandra's auth already has fairly well defined 
>>> interfaces and a plug-in mechanism, so it's easy to vend alternative auth 
>>> solutions without polluting the main project's dependency graph, at 
>>> build-time anyway. A similar approach could be beneficial for CEP-36, 
>>> particularly (IMO) for cold-storage purposes. I suspect decoupling 
>>> pluggable alternate channel proxies for cold storage from configurable 
>>> alternate channel proxies for redirecting data locally to free up space, 
>>> migrate to a different storage device, etc., would make both easier. The 
>>> CEP seems to be trying to do both, but they smell like pretty different 
>>> goals to me.
>>> 
>>> Thanks -- Joel.
>>> 
 
 On Thu, Mar 6, 2025 at 12:53 AM Benedict  wrote:
> I think another way of saying what Stefan may be getting at is what does 
> a library give us that an appropriately config

Re: CEP-15 Update

[Jeremiah Jordan]

I have no stake in this feature besides thinking it would enable some great
new end user workloads. Also it actually makes my professional life easier
the longer this code is not merged.

But given the new feature (and its caveats) are behind a yaml flag, as well
as opt in at the table level, I don’t see a reason to keep it from merging
as is.  If this was not the case I would also be against early merging
before the caveats were resolved.  In my opinion since the feature is self
contained and off by default, I see no issues.

On the topic of “what if they throw it over the fence and run away”. Given
the time invested, and the number of committers and PMC members involved in
the development of the feature, does anyone realistically believe they are
going to drop it in 90% finished and then leave it unfinished?

Do we want to eventually have this feature in the database?  My feeling is
that if we do then we should get it merged to trunk so that it can be
completed faster.

I think we have had this discussion in the past about merging semi-finished
CEPs in stages behind feature flags so that more people could get
visibility into them.  The latest thread I can find doesn’t seem to have
anyone completely against it, with some people wanting to make sure
anything merged did not put trunk into an “un-releasable” state and others
expressing they would want such code to be useful as an MVP.
https://lists.apache.org/thread/c39yrbdszgz9s34vr17wpjdhv6h2oo61

Is someone -1 on this merging early behind the flag?  If not I think it can
be merged whenever? AFAIK all the code in the feature branch has the
required two +1’s already.

-Jeremiah

On Thu, Mar 6, 2025 at 2:44 PM Benedict Elliott Smith 
wrote:

> Because we want to validate against the latest code in trunk, else we are
> validating stale behaviours. The cost of rebasing is high, so we do not do
> it frequently. That means we will likely stop developing OSS-first, as the
> focus will have to move to our internal branch that satisfies these
> criteria.
>
> Exactly what this might be for upstreaming I cannot say. Personally, I aim
> to work exclusively on the branch we are stabilising. If that is not trunk,
> the latency for my contributions being made public might be high, as I have
> a huge imbalance of over-investment to recoup, and anything unnecessary
> will be deferred.
>
> Since the feature is disabled, and the code is almost entirely isolated, I
> cannot imagine the cost to the community to removing this work would be
> very high. But, I do not intend to argue Accord’s case here. I will let you
> all decide.
>
> Please decide soon though, as it shapes our work planning. The positive
> reception so far had lead me to consider prioritising a move to trunk-first
> development within the next week or two, and the associated work that
> entails. However, if that was optimistic we will have to shift our plans.
>
>
>
> On 6 Mar 2025, at 20:16, Jordan West  wrote:
>
> The work and effort in accord has been amazing. And I’m sure it sets a new
> standard for code quality and correctness testing which I’m also entirely
> behind. I also trust the folks working on it want to take it to the a fully
> production ready solution. But I’m worried about circumstances out of our
> control leaving us with a very complex feature that isn’t complete.
>
> I do have some questions. Could folks help me better understand why
> testing real workloads necessitates a merge (my understanding from the
> original reason is this is the impetus for why we would merge now)? Also I
> think the performance and scheme change caveats are rather large ones. One
> of accords promise was better performance and I think making schema changes
> with nodes down not being supported is a big gap. Could we have some
> criteria like “supports all the operations PaxosV2 supports” or “performs
> as well or better than PaxosV2 on [workload(s)]”?
>
> I understand waiting asks a lot of the authors in terms of baring the
> burden of a more complex merge. But I think we also need to consider what
> merging is asking the community to bear if the worst happens and we are
> unable to take the feature from its current state to something that can be
> widely used in production.
>
>
> Jordan
>
>
> On Wed, Mar 5, 2025 at 15:52 Blake Eggleston  wrote:
>
>> +1 to merging it
>>
>> On Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:
>>
>> You have my +1
>>
>> On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:
>> >
>> > Correct, these caveats should only apply to tables that have opted-in
>> to accord.
>> >
>> > On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:
>> >
>> > 
>> > So great to see all this hard work about to pay off!
>> >
>> > On the questions/concerns front, the only concern I would have towards
>> merging this to trunk is if any of the caveats apply when someone is not
>> using Accord.  Assuming they only apply when the feature flag is enabled, I
>> see no reason not to get this merged into trunk once everyo

Re: CEP-15 Update

[Jon Haddad]

Hmm... I took a look at the cep-15-accord branch in GitHub, it looks like
it's several hundred commits behind trunk.  Since you'll need to rebase
again before merge *anyways*, would it make sense to do it once more, and I
can publish easy-cass-lab with the latest branch?  If folks have concerns,
it's easy to fire up a cluster (I do it constantly) and try it out.

I think if we were to do this, out of consideration we should time box the
amount of time for an evaluation and unless someone raises an objection,
consider lazy consensus achieved.

Jon



On Thu, Mar 6, 2025 at 12:46 PM Benedict Elliott Smith 
wrote:

> Because we want to validate against the latest code in trunk, else we are
> validating stale behaviours. The cost of rebasing is high, so we do not do
> it frequently. That means we will likely stop developing OSS-first, as the
> focus will have to move to our internal branch that satisfies these
> criteria.
>
> Exactly what this might be for upstreaming I cannot say. Personally, I aim
> to work exclusively on the branch we are stabilising. If that is not trunk,
> the latency for my contributions being made public might be high, as I have
> a huge imbalance of over-investment to recoup, and anything unnecessary
> will be deferred.
>
> Since the feature is disabled, and the code is almost entirely isolated, I
> cannot imagine the cost to the community to removing this work would be
> very high. But, I do not intend to argue Accord’s case here. I will let you
> all decide.
>
> Please decide soon though, as it shapes our work planning. The positive
> reception so far had lead me to consider prioritising a move to trunk-first
> development within the next week or two, and the associated work that
> entails. However, if that was optimistic we will have to shift our plans.
>
>
>
> On 6 Mar 2025, at 20:16, Jordan West  wrote:
>
> The work and effort in accord has been amazing. And I’m sure it sets a new
> standard for code quality and correctness testing which I’m also entirely
> behind. I also trust the folks working on it want to take it to the a fully
> production ready solution. But I’m worried about circumstances out of our
> control leaving us with a very complex feature that isn’t complete.
>
> I do have some questions. Could folks help me better understand why
> testing real workloads necessitates a merge (my understanding from the
> original reason is this is the impetus for why we would merge now)? Also I
> think the performance and scheme change caveats are rather large ones. One
> of accords promise was better performance and I think making schema changes
> with nodes down not being supported is a big gap. Could we have some
> criteria like “supports all the operations PaxosV2 supports” or “performs
> as well or better than PaxosV2 on [workload(s)]”?
>
> I understand waiting asks a lot of the authors in terms of baring the
> burden of a more complex merge. But I think we also need to consider what
> merging is asking the community to bear if the worst happens and we are
> unable to take the feature from its current state to something that can be
> widely used in production.
>
>
> Jordan
>
>
> On Wed, Mar 5, 2025 at 15:52 Blake Eggleston  wrote:
>
>> +1 to merging it
>>
>> On Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote:
>>
>> You have my +1
>>
>> On Wed, Mar 5, 2025 at 12:16 PM Benedict  wrote:
>> >
>> > Correct, these caveats should only apply to tables that have opted-in
>> to accord.
>> >
>> > On 5 Mar 2025, at 20:08, Jeremiah Jordan  wrote:
>> >
>> > 
>> > So great to see all this hard work about to pay off!
>> >
>> > On the questions/concerns front, the only concern I would have towards
>> merging this to trunk is if any of the caveats apply when someone is not
>> using Accord.  Assuming they only apply when the feature flag is enabled, I
>> see no reason not to get this merged into trunk once everyone involved is
>> happy with the state of it.
>> >
>> > -Jeremiah
>> >
>> > On Mar 5, 2025 at 12:15:23 PM, Benedict Elliott Smith <
>> bened...@apache.org> wrote:
>> >>
>> >> That depends on all of you lovely people :D
>> >>
>> >> I think we should have finished merging everything we want before QA
>> by ~Monday; certainly not much later.
>> >>
>> >> I think we have some upgrade and python dtest failures to address as
>> well.
>> >>
>> >> So it could be pretty soon if the community is supportive.
>> >>
>> >> On 5 Mar 2025, at 17:22, Patrick McFadin  wrote:
>> >>
>> >>
>> >> What is the timing for starting the merge process? I'm asking because
>> >>
>> >> I have (yet another) presentation and this would be a cool update.
>> >>
>> >>
>> >> On Wed, Mar 5, 2025 at 1:22 AM Benedict Elliott Smith
>> >>
>> >>  wrote:
>> >>
>> >> >
>> >>
>> >> > Thanks everyone.
>> >>
>> >> >
>> >>
>> >> > Jon - your help will be greatly appreciated. We’ll let you know when
>> we’ve got the cycles to invest in performance work (hopefully fairly soon).
>> I expect the first step will be improving visibility 

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Joel Shepherd]

On 3/6/2025 7:16 AM, Jon Haddad wrote:
Assuming everything else is identical, might not matter for S3. 
However, not every object store has a filesystem mount.


Regarding sprawling dependencies, we can always make the provider 
specific libraries available as a separate download and put them on 
their own thread with a separate class path. I think in JVM dtest does 
this already.  Someone just started asking about IAM for login, it 
sounds like a similar problem.


That was me. :-) Cassandra's auth already has fairly well defined 
interfaces and a plug-in mechanism, so it's easy to vend alternative 
auth solutions without polluting the main project's dependency graph, at 
build-time anyway. A similar approach could be beneficial for CEP-36, 
particularly (IMO) for cold-storage purposes. I suspect decoupling 
pluggable alternate channel proxies for cold storage from configurable 
alternate channel proxies for redirecting data locally to free up space, 
migrate to a different storage device, etc., would make both easier. The 
CEP seems to be trying to do both, but they smell like pretty different 
goals to me.


Thanks -- Joel.



On Thu, Mar 6, 2025 at 12:53 AM Benedict  wrote:

I think another way of saying what Stefan may be getting at is
what does a library give us that an appropriately configured mount
dir doesn’t?

We don’t want to treat S3 the same as local disk, but this can be
achieved easily with config. Is there some other benefit of direct
integration? Well defined exceptions if we need to distinguish
cases is one that maybe springs to mind but perhaps there are others?



On 6 Mar 2025, at 08:39, Štefan Miklošovič
 wrote:


That is cool but this still does not show / explain how it would
look like when it comes to dependencies needed for actually
talking to storages like s3.

Maybe I am missing something here and please explain when I am
mistaken but If I understand that correctly, for talking to s3 we
would need to use a library like this, right? (1). So that would
be added among Cassandra dependencies? Hence Cassandra starts to
be biased against s3? Why s3? Every time somebody comes up with a
new remote storage support, that would be added to classpath as
well? How are these dependencies going to play with each other
and with Cassandra in general? Will all these storage
provider libraries for arbitrary clouds be even compatible with
Cassandra licence-wise?

I am sorry I keep repeating these questions but this part of that
I just don't get at all.

We can indeed add an API for this, sure sure, why not. But for
people who do not want to deal with this at all and just be OK
with a FS mounted, why would we block them doing that?

(1)
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/pom.xml

On Wed, Mar 5, 2025 at 3:28 PM Mick Semb Wever 
wrote:

   .

It’s not an area where I can currently dedicate
engineering effort. But if others are interested in
contributing a feature like this, I’d see it as valuable
for the project and would be happy to collaborate on
design/architecture/goals.



Jake mentioned 17 months ago a custom FileSystemProvider we
could offer.

None of us at DataStax has gotten around to providing that,
but to quickly throw something over the wall this is it:

https://github.com/datastax/cassandra/blob/main/src/java/org/apache/cassandra/io/storage/StorageProvider.java

  (with a few friend classes under o.a.c.io.util)

We then have a RemoteStorageProvider, private in another
repo, that implements that and also provides the
RemoteFileSystemProvider that Jake refers to.

Hopefully that's a start to get people thinking about CEP
level details, while we get a cleaned abstract of
RemoteStorageProvider and friends to offer.

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Benedict]

It anyway seems reasonable to me that we would support multiple FileSystemProvider. So perhaps this is really two problems we’re maybe conflating: 1) a mechanism for dropping jars that can register a FileSystemProvider for Cassandra to utilise2) a way to mark directories (from any provider) as “remote storage” so that they can be treated in the appropriate mannerI think the harder problems by far live in (2). Less for performance, but more how we handle errors correctly. For instance, a failure to read from object storage may mean that the data is lost or it may mean the service has been interrupted. This might mean a total or partial loss of any intersecting tokens, depending on how the cluster stripes dependency on the object storage. But we almost certainly don’t want to handle this like we do local disk errors, either way.On 6 Mar 2025, at 15:16, Jon Haddad  wrote:Assuming everything else is identical, might not matter for S3. However, not every object store has a filesystem mount. Regarding sprawling dependencies, we can always make the provider specific libraries available as a separate download and put them on their own thread with a separate class path. I think in JVM dtest does this already.  Someone just started asking about IAM for login, it sounds like a similar problem. On Thu, Mar 6, 2025 at 12:53 AM Benedict  wrote:I think another way of saying what Stefan may be getting at is what does a library give us that an appropriately configured mount dir doesn’t?We don’t want to treat S3 the same as local disk, but this can be achieved easily with config. Is there some other benefit of direct integration? Well defined exceptions if we need to distinguish cases is one that maybe springs to mind but perhaps there are others?On 6 Mar 2025, at 08:39, Štefan Miklošovič  wrote:That is cool but this still does not show / explain how it would look like when it comes to dependencies needed for actually talking to storages like s3. Maybe I am missing something here and please explain when I am mistaken but If I understand that correctly, for talking to s3 we would need to use a library like this, right? (1). So that would be added among Cassandra dependencies? Hence Cassandra starts to be biased against s3? Why s3? Every time somebody comes up with a new remote storage support, that would be added to classpath as well? How are these dependencies going to play with each other and with Cassandra in general? Will all these storage provider libraries for arbitrary clouds be even compatible with Cassandra licence-wise?I am sorry I keep repeating these questions but this part of that I just don't get at all. We can indeed add an API for this, sure sure, why not. But for people who do not want to deal with this at all and just be OK with a FS mounted, why would we block them doing that?(1) https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/pom.xmlOn Wed, Mar 5, 2025 at 3:28 PM Mick Semb Wever  wrote:   .  It’s not an area where I can currently dedicate engineering effort. But if others are interested in contributing a feature like this, I’d see it as valuable for the project and would be happy to collaborate on design/architecture/goals.Jake mentioned 17 months ago a custom FileSystemProvider we could offer.None of us at DataStax has gotten around to providing that, but to quickly throw something over the wall this is it: https://github.com/datastax/cassandra/blob/main/src/java/org/apache/cassandra/io/storage/StorageProvider.java   (with a few friend classes under o.a.c.io.util)We then have a RemoteStorageProvider, private in another repo, that implements that and also provides the RemoteFileSystemProvider that Jake refers to.Hopefully that's a start to get people thinking about CEP level details, while we get a cleaned abstract of RemoteStorageProvider and friends to offer.

Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations

[Jon Haddad]

Hey Joel, thanks for chiming in!

Regarding dependencies - while it's possible to provide pluggable
interfaces, the issue I'm concerned about is conflicting versions of
transitive dependencies at runtime.  For example, I used a java agent that
had a different version of snakeyaml, and it ended up breaking C*'s startup
sequence [1].  I suggest putting external modules on separate threads with
their own classpath to avoid this issue.

I think there's quite a bit of overlap between the two desires expressed in
this thread, even though they achieve very different results.  I personally
can't see myself using something that treats an object store as cold
storage where SSTables are moved (implying they weren't there before), and
I've expressed my concerns with this, but other folks seem to want it and
that's OK.  I feel very strongly that treating local storage as a cache
with the full dataset on object store is a better approach, but ultimately
different people have different priorities.  Either way, stuff is moved to
object store at some point, and pulled to the local disk on demand.

I am *firmly* of the position that this CEP should not exclude the local
storage as cache option, and should be accounted for in the design.

Jon

[1] https://issues.apache.org/jira/browse/CASSANDRA-19663


On Thu, Mar 6, 2025 at 10:31 AM Joel Shepherd  wrote:

> On 3/6/2025 7:16 AM, Jon Haddad wrote:
>
> Assuming everything else is identical, might not matter for S3. However,
> not every object store has a filesystem mount.
>
> Regarding sprawling dependencies, we can always make the provider specific
> libraries available as a separate download and put them on their own thread
> with a separate class path. I think in JVM dtest does this already.
> Someone just started asking about IAM for login, it sounds like a similar
> problem.
>
> That was me. :-) Cassandra's auth already has fairly well defined
> interfaces and a plug-in mechanism, so it's easy to vend alternative auth
> solutions without polluting the main project's dependency graph, at
> build-time anyway. A similar approach could be beneficial for CEP-36,
> particularly (IMO) for cold-storage purposes. I suspect decoupling
> pluggable alternate channel proxies for cold storage from configurable
> alternate channel proxies for redirecting data locally to free up space,
> migrate to a different storage device, etc., would make both easier. The
> CEP seems to be trying to do both, but they smell like pretty different
> goals to me.
>
> Thanks -- Joel.
>
>
> On Thu, Mar 6, 2025 at 12:53 AM Benedict  wrote:
>
>> I think another way of saying what Stefan may be getting at is what does
>> a library give us that an appropriately configured mount dir doesn’t?
>>
>> We don’t want to treat S3 the same as local disk, but this can be
>> achieved easily with config. Is there some other benefit of direct
>> integration? Well defined exceptions if we need to distinguish cases is one
>> that maybe springs to mind but perhaps there are others?
>>
>>
>> On 6 Mar 2025, at 08:39, Štefan Miklošovič 
>> wrote:
>>
>> 
>>
>> That is cool but this still does not show / explain how it would look
>> like when it comes to dependencies needed for actually talking to storages
>> like s3.
>>
>> Maybe I am missing something here and please explain when I am mistaken
>> but If I understand that correctly, for talking to s3 we would need to use
>> a library like this, right? (1). So that would be added among Cassandra
>> dependencies? Hence Cassandra starts to be biased against s3? Why s3? Every
>> time somebody comes up with a new remote storage support, that would be
>> added to classpath as well? How are these dependencies going to play with
>> each other and with Cassandra in general? Will all these storage
>> provider libraries for arbitrary clouds be even compatible with Cassandra
>> licence-wise?
>>
>> I am sorry I keep repeating these questions but this part of that I just
>> don't get at all.
>>
>> We can indeed add an API for this, sure sure, why not. But for people who
>> do not want to deal with this at all and just be OK with a FS mounted, why
>> would we block them doing that?
>>
>> (1)
>> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/pom.xml
>>
>> On Wed, Mar 5, 2025 at 3:28 PM Mick Semb Wever  wrote:
>>
>>>.
>>>
>>>
>>> It’s not an area where I can currently dedicate engineering effort. But
 if others are interested in contributing a feature like this, I’d see it as
 valuable for the project and would be happy to collaborate on
 design/architecture/goals.

>>>
>>>
>>> Jake mentioned 17 months ago a custom FileSystemProvider we could offer.
>>>
>>> None of us at DataStax has gotten around to providing that, but to
>>> quickly throw something over the wall this is it:
>>>
>>> https://github.com/datastax/cassandra/blob/main/src/java/org/apache/cassandra/io/storage/StorageProvider.java
>>>
>>>   (with a few friend classes under o.a.c.io.util)
>>>
>>> We 

[DISCUSS] Plugins and dependencies

[Joel Shepherd]

Splitting this out from the CEP-36 thread.

I agree: dependency collisions at run-time are a problem. It's made even 
worse by the possibility of users using multiple plugins (authn, authz, 
compression, storage, etc.).


It also cuts two ways. E.g. the interfaces that plugin authenticators 
need to implement are defined in org.apache.cassandra.auth, so as far as 
I know the plugin has to take a build-time dependency on the main 
Cassandra module itself, and pull in all of its dependencies. (I'd love 
to be told that I'm mistaken.) In addition to the risk of version 
conflicts, it increases the risk of a change to Cassandra's own 
dependencies inadvertently breaking a plugin that's taken a transitive 
dependency. Might be bad form on the plugin's part, but certainly possible.


I've gotten the impression that there's not a lot of enthusiasm for 
breaking apart the main Cassandra module, but I have wondered if it'd be 
worth making an exception for the interfaces plugins are supposed to 
code against. It'd be nice to depend on those without pulling in the 
rest of the project, and it'd be another step towards reducing the risk 
of plugins breaking because of dependency changes in the main project.


-- Joel.

On 3/6/2025 10:52 AM, Jon Haddad wrote:

Hey Joel, thanks for chiming in!

Regarding dependencies - while it's possible to provide pluggable 
interfaces, the issue I'm concerned about is conflicting versions of 
transitive dependencies at runtime. For example, I used a java agent 
that had a different version of snakeyaml, and it ended up breaking 
C*'s startup sequence [1].  I suggest putting external modules on 
separate threads with their own classpath to avoid this issue.


I think there's quite a bit of overlap between the two desires 
expressed in this thread, even though they achieve very different 
results.  I personally can't see myself using something that treats an 
object store as cold storage where SSTables are moved (implying they 
weren't there before), and I've expressed my concerns with this, but 
other folks seem to want it and that's OK.  I feel very strongly that 
treating local storage as a cache with the full dataset on object 
store is a better approach, but ultimately different people have 
different priorities.  Either way, stuff is moved to object store at 
some point, and pulled to the local disk on demand.


I am *firmly* of the position that this CEP should not exclude the 
local storage as cache option, and should be accounted for in the design.


Jon

[1] https://issues.apache.org/jira/browse/CASSANDRA-19663


On Thu, Mar 6, 2025 at 10:31 AM Joel Shepherd  wrote:

On 3/6/2025 7:16 AM, Jon Haddad wrote:

Assuming everything else is identical, might not matter for S3.
However, not every object store has a filesystem mount.

Regarding sprawling dependencies, we can always make the provider
specific libraries available as a separate download and put them
on their own thread with a separate class path. I think in JVM
dtest does this already.  Someone just started asking about IAM
for login, it sounds like a similar problem.


That was me. :-) Cassandra's auth already has fairly well defined
interfaces and a plug-in mechanism, so it's easy to vend
alternative auth solutions without polluting the main project's
dependency graph, at build-time anyway. A similar approach could
be beneficial for CEP-36, particularly (IMO) for cold-storage
purposes. I suspect decoupling pluggable alternate channel proxies
for cold storage from configurable alternate channel proxies for
redirecting data locally to free up space, migrate to a different
storage device, etc., would make both easier. The CEP seems to be
trying to do both, but they smell like pretty different goals to me.

Thanks -- Joel.



On Thu, Mar 6, 2025 at 12:53 AM Benedict  wrote:

I think another way of saying what Stefan may be getting at
is what does a library give us that an appropriately
configured mount dir doesn’t?

We don’t want to treat S3 the same as local disk, but this
can be achieved easily with config. Is there some other
benefit of direct integration? Well defined exceptions if we
need to distinguish cases is one that maybe springs to mind
but perhaps there are others?



On 6 Mar 2025, at 08:39, Štefan Miklošovič
 wrote:


That is cool but this still does not show / explain how it
would look like when it comes to dependencies needed for
actually talking to storages like s3.

Maybe I am missing something here and please explain when I
am mistaken but If I understand that correctly, for talking
to s3 we would need to use a library like this, right? (1).
So that would be added among Cassandra dependencies? Hence
Cassandra starts to be biased against s3? Why s3? Every time
somebod

Re: [DISCUSS] AWS IAM-based client authentication

[Joel Shepherd]

Thanks Scott - That gives me a clear to direction to go in -- Joel.

On 3/5/2025 9:01 PM, C. Scott Andreas wrote:


Joel, thanks for reaching out.

This sounds interesting, I bet there are many who would benefit from 
IAM-based authentication.


If you haven't yet, could you request a Jira account? Someone will be 
able to approve it almost immediately if you don't have one yet. 
https://selfserve.apache.org/jira-account.html


For discussing/reviewing the implementation, I'd make the repos public 
and create a ticket under the database [1] and driver [2] projects 
with a description and source link to start.


For new feature proposals, we'll usually open with a discuss thread as 
you've started here. That discussion will gauge receptivity and 
whether to proceed by acclamation; or whether the proposal is 
significant enough in scope to warrant a CEP doc and vote thread [3].


Cheers,

– Scott

[1] http://issues.apache.org/jira/browse/CASSANDRA
[2] http://issues.apache.org/jira/browse/CASSJAVA
[3] 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201




On Mar 4, 2025, at 12:48 PM, Joel Shepherd  wrote:


Hi - I have a side project that provides client- and node-side Java
plug-ins to enable client-to-node authentication based on AWS
identities. This would, for example, enable clients to use EC2 instance
roles to authenticate to Cassandra nodes, or use ordinary IAM
keys/secret keys. The client needs to be able to obtain valid IAM
credentials to sign a request, and the node needs to be able to connect
to a public AWS Security Token Service (STS) endpoint. There are no
other required AWS dependencies, and (I believe) no changes required
driver or node code: just minor configuration updates.

I'm seeking help in reviewing the concept and code. I'm new to this
community,  so I'm looking for suggestions on how to best engage you on
this.

The code (which is not quite production-ready) is in two private GitHub
repos which I'm happy to grant access to for early review. I can also
provide documentation on the approach: not sure whether that's best
shared via this thread, a CEP, repo documentation ... suggestions wanted.

Thanks: I'd appreciate any and all help in making these plug-ins
available to the community.

-- Joel.