Re: [DISCUSSION] New dependencies for SAI CEP-7

[Caleb Rackliffe]

We need random generators no matter what for these tests, so I think what
we need to decide is whether to continue to use Carrot or migrate those to
QuickTheories, along the lines of what we have now in
org.apache.cassandra.utils.Generators.

When it comes to a library like this, the thing I would optimize for is how
much it already provides (and therefore how much we need to write and
maintain ourselves). If you look at something like NumericTypeSortingTest
in the 18058 branch , it's
pretty compact w/ Carrot's RandomizedTest in use, but I suppose it could
also use IntegersDSL from QT...

(Not that it matters, but just for reference, we do use
com.carrotsearch.hppc already.)

On Tue, Dec 13, 2022 at 10:14 AM Mike Adamson  wrote:

> Can you talk more about why?  There are several ways to do random testing
>> in-tree ATM, so wondering why we need another one
>
>
> I can see one mechanism for random testing in-tree. That is the Simulator
> but that seems primarily involved in the random orchestration of
> operations. My apologies if I have simplified its significance. Apart from
> that, I can only see different usages of Random in unit tests. I admit I
> have not looked beyond this at dtests.
>
> The random testing in SAI is more focussed on the behaviour of the
> low-level index structures and flow of data to / from these. Using randomly
> generated values in tests has proved invaluable in highlighting edge
> conditions in the code. This above library was only added to provide us
> with a rich set of random generators. I am happy to look at removing this
> library if its inclusion is contentious.
>
>
> On Mon, 12 Dec 2022 at 19:41, David Capwell  wrote:
>
>> com.carrotsearch.randomizedtesting.randomizedtesting-runner 2.1.2 - test
>> dependency
>>
>>
>> Can you talk more about why?  There are several ways to do random testing
>> in-tree ATM, so wondering why we need another one
>>
>>
>> On Dec 8, 2022, at 6:51 AM, Mike Adamson  wrote:
>>
>> Hi,
>>
>> I wanted to discuss the addition of the following dependencies for CEP-7.
>> The dependencies are:
>>
>> org.apache.lucene.lucene-core 7.5.0
>> org.apache.lucene.lucene-analyzers-common 7.5.0
>> com.carrotsearch.randomizedtesting.randomizedtesting-runner 2.1.2 - test
>> dependency
>>
>> Lucene is an apache project so is licensed APL2. Carrotsearch is not an
>> apache project but is licensed APL2
>>
>> We are also removing the dependency
>> on com.github.rholder.snowball-stemmer. This library is used by SASI
>> stemming filters but a later version of the same library is available in
>> the lucene libraries.
>>
>> Does anyone have any concerns about these changes?
>>
>> Mike Adamson
>>
>>
>>
>
> --
> [image: DataStax Logo Square]  *Mike Adamson*
> Engineering
>
> +1 650 389 6000 <16503896000> | datastax.com 
> Find DataStax Online: [image: LinkedIn Logo]
> 
>[image: Facebook Logo]
> 
>[image: Twitter Logo]    [image: RSS
> Feed]    [image: Github Logo]
> 
>
>

Re: Cassandra CI Status 2023-01-07

[Caleb Rackliffe]

New failures from Build Lead Week 4:

*** CASSANDRA-18188 - Test failure in
upgrade_tests.cql_tests.cls.test_limit_ranges
- trunk
- AttributeError: module 'py' has no attribute 'io'

*** CASSANDRA-18189 - Test failure in
cqlsh_tests.test_cqlsh_copy.TestCqlshCopy.test_bulk_round_trip_with_timeouts
- 4.0
- assert 10 == 94764
- other failures currently open in this test class, but at least
superficially, different errors (see CASSANDRA-17322, CASSANDRA-18162)

Timeouts continue to manifest in many places.

On Sun, Jan 15, 2023 at 6:02 AM Mick Semb Wever  wrote:

> *** The Butler (Build Lead)
>>
>> The introduction of Butler and the Build Lead was a wonderful
>> improvement to our CI efforts.  It has brought a lot of hygiene in
>> listing out flakies as they happened.  Noted that this has in-turn
>> increased the burden in getting our major releases out, but that's to
>> be seen as a one-off cost.
>>
>
>
> New Failures from Build Lead Week 3.
>
>
> *** CASSANDRA-18156
> – 
> repair_tests.deprecated_repair_test.TestDeprecatedRepairNotifications.test_deprecated_repair_error_notification
>  - AssertionError: Node logs don't have an error message for the failed
> repair
>  - hard regression
>  - 3.0, 3.11,
>
> *** CASSANDRA-18164 – CASTest Message serializedSize(12) does not match
> what was written with serialize(out, 12) for verb
> PAXOS2_COMMIT_AND_PREPARE_RSP
>  - serializer class org.apache.cassandra.net.Message$Serializer; expected
> 1077, actual 1079
>  - 4.1, trunk
>
> *** CASSANDRA-18158
> – 
> org.apache.cassandra.distributed.upgrade.MixedModeReadTest.mixedModeReadColumnSubsetDigestCheck
>  - Cannot achieve consistency level ALL
>  - 3.11, trunk
>
> *** CASSANDRA-18159 – repair_tests.repair_test.TestRepair.test_*dc_repair
>   - AssertionError: null
> in MemtablePool$SubPool.released(MemtablePool.java:193)
>  - 3.11, 4.0, 4.1, trunk
>
> *** CASSANDRA-18160
> – 
> cdc_test.TestCDC.test_insertion_and_commitlog_behavior_after_reaching_cdc_total_space
>  - Found orphaned index file in after CDC state not in former
>  - 4.1, trunk
>
> *** CASSANDRA-18161 –
>  
> org.apache.cassandra.transport.CQLConnectionTest.handleCorruptionOfLargeMessageFrame
>  - AssertionFailedError in
> CQLConnectionTest.testFrameCorruption(CQLConnectionTest.java:491)
>  - 4.0, 4.1, trunk
>
> *** CASSANDRA-18162 –
> cqlsh_tests.test_cqlsh_copy.TestCqlshCopy.test_bulk_round_trip_non_prepared_statements
> - Inet address 127.0.0.3:7000 is not available: [Errno 98] Address
> already in use
> - 3.0, 3.11, 4.0, 4.1, trunk
>
> *** CASSANDRA-18163 –
>  
> transient_replication_test.TestTransientReplicationRepairLegacyStreaming.test_speculative_write_repair_cycle
>  - AssertionError Incoming stream entireSSTable
>  - 4.0, 4.1, trunk
>
>
> While writing these up, some thoughts…
>  - While Butler reports failures against multiple branches, there's no
> feedback/sync that the ticket needs its fixVersions updated when failures
> happen in other branches after the ticket is created.
>  - In 4.0 onwards, a majority of the failures are timeouts (>900s),
> reinforcing that the current main problem we are facing in ci-cassandra.a.o
> is saturation/infra
>
>
>
>
>

Re: Merging CEP-15 to trunk

[Caleb Rackliffe]

Just FYI, I'm going to be posting a Jira (which will have some dependencies
as well) to track this merge, hopefully some time today...

On Tue, Jan 24, 2023 at 12:26 PM Ekaterina Dimitrova 
wrote:

> I actually see people all the time making a final check before merge as
> part of the review. And I personally see it only as a benefit when it comes
> to serious things like Accord, as an example. Even as a help for the author
> who is overwhelmed with the big amount of work already done - someone to do
> quick last round of review. Team work after all.
>
> Easy rebase - those are great news. I guess any merge conflicts that were
> solved will be documented and confirmed with reviewers before merge on the
> ticket where the final CI push will be posted. I also assumed that even
> without direct conflicts a check that there is no contradiction with any
> post-September commits is done as part of the rebase. (Just adding here for
> completeness)
>
> One thing that I wanted to ask for is when you push to CI, you or whoever
> does it, to approve all jobs. Currently we have pre-approved the minimum
> required jobs in the pre-commit workflow. I think in this case with a big
> work approving all jobs in CircleCI will be of benefit. (I also do it for
> bigger bodies of work to be on the safe side) Just pointing in case you use
> a script or something to push only the pre-approved ones. Please ping me in
> Slack if It’s not clear what I mean, happy to help with that
>
> On Tue, 24 Jan 2023 at 11:52, Benedict  wrote:
>
>> Perhaps the disconnect is that folk assume a rebase will be difficult and
>> have many conflicts?
>>
>> We have introduced primarily new code with minimal integration points, so
>> I decided to test this. I managed to rebase locally in around five minutes;
>> mostly imports. This is less work than for a rebase of fairly typical
>> ticket of average complexity.
>>
>> Green CI is of course a requirement. There is, however, no good
>> procedural or technical justification for a special review of the rebase.
>>
>> Mick is encouraged to take a look at the code before and after rebase,
>> and will be afforded plenty of time to do so. But I will not gate merge on
>> this adhoc requirement.
>>
>>
>>
>> On 24 Jan 2023, at 15:40, Ekaterina Dimitrova 
>> wrote:
>>
>> 
>>
>> Hi everyone,
>> I am excited to see this work merged. I noticed the branch is 395 commits
>> behind trunk or not rebased since September last year. I think if Mick or
>> anyone else wants to make a final pass after rebase happens and CI runs -
>> this work can only benefit of that. Squash, rebase and full CI run green is
>> the minimum that, if I read correctly the thread, we all agree on that
>> part.
>> I would say that CI and final check after a long rebase of a patch is a
>> thing we actually do all the time even for small patches when we get back
>> to our backlog of old patches. This doesn’t mean that the previous reviews
>> are dismissed or people not trusted or anything like that.
>> But considering the size and the importance of this work, I can really
>> see only benefit of a final cross-check.
>> As Henrik mentioned me, I am not sure I will have the chance to review
>> this work any time soon (just setting the right expectations up front) but
>> I see at least Mick already mentioning he would do it if there are no other
>> volunteers. Now, whether it will be separate ticket or not, that is a
>> different story. Aren’t the Accord tickets in an epic under which we can
>> document the final rebase, CI runs, etc?
>>
>> On Tue, 24 Jan 2023 at 9:40, Henrik Ingo 
>> wrote:
>>
>>> When was the last time the feature branch was rebased? Assuming it's a
>>> while back and the delta is significant, surely it's normal process to
>>> first rebase, run tests, and then present the branch for review?
>>>
>>> To answer your question: The effect of the rebase is then either baked
>>> into the original commits (which I personally dislike), or you can also
>>> have the rebase-induced changes as their own commits. (Which can get
>>> tedious, but has the benefit of making explicit what was only a change due
>>> to rebasing.) Depending on which approach you take when rebasing, a
>>> reviewer would then review accordingly.
>>>
>>> henrik
>>>
>>> On Tue, Jan 24, 2023 at 11:14 AM Benedict  wrote:
>>>
 No, that is not the normal process. What is it you think you would be
 reviewing? There are no diffs produced as part of rebasing, and the purpose
 of review is to ensure code meets out standards, not that the committer is
 competent at rebasing or squashing. Nor are you familiar with the code as
 it was originally reviewed, so would have nothing to compare against. We
 expect a clean CI run, ordinarily, not an additional round of review. If we
 were to expect that, it would be by the original reviewer, not a third
 party, as they are the only ones able to judge the rebase efficiently.

 I would support investigating too

Re: Merging CEP-15 to trunk

[Caleb Rackliffe]

might be treating Ninja commits a bit differently on a feature
>> branch than trunk, which intersects with 1 and 2
>>
>> My personal opinions are:
>> I disagree with 1 - it simply takes the added effort of actively
>> following that branch and respective JIRAs if you're interested. I think
>> having a feature branch in the ASF git repo w/commits and JIRAs tracking
>> that work is perfectly fine, and the existing bar (2 committers +1, green
>> tests before merge to trunk) when applied to a feature branch is also not
>> just well within the "letter of the law" on the project but also logically
>> sufficient to retain our bar of quality and stability.
>>
>> For 2 (reviews required after rebase) I don't think we should
>> over-prescribe process on this. If all tests are green pre-rebase, and all
>> tests are green post-rebase, and a committer is confident they didn't
>> materially modify the functioning of the logical flow or data structures of
>> their code during a rebase, I don't see there being any value added by
>> adding another review based on those grounds.
>>
>> If the subtext is actually that some folks feel we need a discussion
>> about whether we should have a different bar for review on CEP feature
>> branches (3 committers? 1+ pmc members? more diversity in reviewers or
>> committers as measured by some as yet unspoken metric), perhaps we could
>> have that discussion. FWIW I'm against changes there as well; we all wear
>> our Apache Hats here, and if the debate is between work like this happening
>> in a feature branch affording contributors increased efficiency and
>> locality vs. all that happening on trunk and repeatedly colliding with
>> everyone everywhere, feature branches are a clear win IMO.
>>
>> And for 3 - I think we've all broadly agreed we shouldn't ninja commit
>> unless it's a comment fix, typo, forgotten git add, or something along
>> those lines. For any commit that doesn't qualify it should go through the
>> review process.
>>
>> And a final note - Ekaterina alluded to something valuable in her email
>> earlier in this thread. I think having a "confirm green on all the test
>> suites that are green on merge target" bar for large feature branches
>> (rather than strictly the "pre-commit subset") before merge makes a lot of
>> sense.
>>
>> On Tue, Jan 24, 2023, at 1:41 PM, Caleb Rackliffe wrote:
>>
>> Just FYI, I'm going to be posting a Jira (which will have some
>> dependencies as well) to track this merge, hopefully some time today...
>>
>> On Tue, Jan 24, 2023 at 12:26 PM Ekaterina Dimitrova <
>> e.dimitr...@gmail.com> wrote:
>>
>> I actually see people all the time making a final check before merge as
>> part of the review. And I personally see it only as a benefit when it comes
>> to serious things like Accord, as an example. Even as a help for the author
>> who is overwhelmed with the big amount of work already done - someone to do
>> quick last round of review. Team work after all.
>>
>> Easy rebase - those are great news. I guess any merge conflicts that were
>> solved will be documented and confirmed with reviewers before merge on the
>> ticket where the final CI push will be posted. I also assumed that even
>> without direct conflicts a check that there is no contradiction with any
>> post-September commits is done as part of the rebase. (Just adding here for
>> completeness)
>>
>> One thing that I wanted to ask for is when you push to CI, you or whoever
>> does it, to approve all jobs. Currently we have pre-approved the minimum
>> required jobs in the pre-commit workflow. I think in this case with a big
>> work approving all jobs in CircleCI will be of benefit. (I also do it for
>> bigger bodies of work to be on the safe side) Just pointing in case you use
>> a script or something to push only the pre-approved ones. Please ping me in
>> Slack if It’s not clear what I mean, happy to help with that
>>
>> On Tue, 24 Jan 2023 at 11:52, Benedict  wrote:
>>
>>
>> Perhaps the disconnect is that folk assume a rebase will be difficult and
>> have many conflicts?
>>
>> We have introduced primarily new code with minimal integration points, so
>> I decided to test this. I managed to rebase locally in around five minutes;
>> mostly imports. This is less work than for a rebase of fairly typical
>> ticket of average complexity.
>>
>> Green CI is of course a requirement. There is, however, no good
>> procedural or technica

Re: Merging CEP-15 to trunk

[Caleb Rackliffe]

with multiple people collaborating
> on a feature branch like this. This discussion seems to have surfaced a few
> sentiments:
>
> 1. Some contributors seem to feel that work on a feature branch doesn't
> have the same inherent visibility as work on trunk
> 2. There's a lack of clarity on our review process when it comes to
> significant (in either time or size) rebases
> 3. We might be treating Ninja commits a bit differently on a feature
> branch than trunk, which intersects with 1 and 2
>
> My personal opinions are:
> I disagree with 1 - it simply takes the added effort of actively following
> that branch and respective JIRAs if you're interested. I think having a
> feature branch in the ASF git repo w/commits and JIRAs tracking that work
> is perfectly fine, and the existing bar (2 committers +1, green tests
> before merge to trunk) when applied to a feature branch is also not just
> well within the "letter of the law" on the project but also logically
> sufficient to retain our bar of quality and stability.
>
> For 2 (reviews required after rebase) I don't think we should
> over-prescribe process on this. If all tests are green pre-rebase, and all
> tests are green post-rebase, and a committer is confident they didn't
> materially modify the functioning of the logical flow or data structures of
> their code during a rebase, I don't see there being any value added by
> adding another review based on those grounds.
>
> If the subtext is actually that some folks feel we need a discussion about
> whether we should have a different bar for review on CEP feature branches
> (3 committers? 1+ pmc members? more diversity in reviewers or committers as
> measured by some as yet unspoken metric), perhaps we could have that
> discussion. FWIW I'm against changes there as well; we all wear our Apache
> Hats here, and if the debate is between work like this happening in a
> feature branch affording contributors increased efficiency and locality vs.
> all that happening on trunk and repeatedly colliding with everyone
> everywhere, feature branches are a clear win IMO.
>
> And for 3 - I think we've all broadly agreed we shouldn't ninja commit
> unless it's a comment fix, typo, forgotten git add, or something along
> those lines. For any commit that doesn't qualify it should go through the
> review process.
>
> And a final note - Ekaterina alluded to something valuable in her email
> earlier in this thread. I think having a "confirm green on all the test
> suites that are green on merge target" bar for large feature branches
> (rather than strictly the "pre-commit subset") before merge makes a lot of
> sense.
>
> On Tue, Jan 24, 2023, at 1:41 PM, Caleb Rackliffe wrote:
>
> Just FYI, I'm going to be posting a Jira (which will have some
> dependencies as well) to track this merge, hopefully some time today...
>
> On Tue, Jan 24, 2023 at 12:26 PM Ekaterina Dimitrova <
> e.dimitr...@gmail.com> wrote:
>
> I actually see people all the time making a final check before merge as
> part of the review. And I personally see it only as a benefit when it comes
> to serious things like Accord, as an example. Even as a help for the author
> who is overwhelmed with the big amount of work already done - someone to do
> quick last round of review. Team work after all.
>
> Easy rebase - those are great news. I guess any merge conflicts that were
> solved will be documented and confirmed with reviewers before merge on the
> ticket where the final CI push will be posted. I also assumed that even
> without direct conflicts a check that there is no contradiction with any
> post-September commits is done as part of the rebase. (Just adding here for
> completeness)
>
> One thing that I wanted to ask for is when you push to CI, you or whoever
> does it, to approve all jobs. Currently we have pre-approved the minimum
> required jobs in the pre-commit workflow. I think in this case with a big
> work approving all jobs in CircleCI will be of benefit. (I also do it for
> bigger bodies of work to be on the safe side) Just pointing in case you use
> a script or something to push only the pre-approved ones. Please ping me in
> Slack if It’s not clear what I mean, happy to help with that
>
> On Tue, 24 Jan 2023 at 11:52, Benedict  wrote:
>
>
> Perhaps the disconnect is that folk assume a rebase will be difficult and
> have many conflicts?
>
> We have introduced primarily new code with minimal integration points, so
> I decided to test this. I managed to rebase locally in around five minutes;
> mostly imports. This is less work than for a rebase of fairly typical
> ticket of average complexity.
>
> Gre

Re: Merging CEP-15 to trunk

[Caleb Rackliffe]

Just an FYI, the Accord feature flag has landed in the cep-15-accord
branch:
https://github.com/apache/cassandra/commit/2e680a33c03ce66d4b1358e1a1cc11cf4ee0189f

(btw, it implicitly fixes some of the dtests around the new Accord system
keyspace, because Accord is now disabled by default.)

On Tue, Jan 31, 2023 at 3:08 PM Ekaterina Dimitrova 
wrote:

> Thanks David, I think I am personally aligned with what you are saying.
> Thanks for clarifying your previous email.
>
> “ I do not believe anyone is arguing to merge Accord with bugs… bugs we
> know about are blockers for merge.  There are pending improvements and
> features people are working on, but I don’t believe I know of us trying to
> merge bugs into trunk.  If there are bugs found we should address before
> merge, but improvements can happen after trunk merge (depending on timing,
> before merge is fine).
>
> I do hope this comment doesn’t cause a rabbit hole of “what is a bug vs
> what is an improvement”….”
>
> No doubts on my end personally
>
> Excited to see Accord landing :-)
>
> On Tue, 31 Jan 2023 at 15:35, David Capwell  wrote:
>
>> About this merge - I would  personally love to see everything rebased and
>> things rerun in CI which no one has a doubt it will happen
>>
>>
>> Yes, I think this is expected behavior that must happen before adding to
>> trunk, or even just rebasing the feature branch; CI must be passing and
>> flaky tests (as you called out) must be resolved.  Nothing is special from
>> a process point of view as this is already the expected process.  I know
>> Caleb is working on it and uses my circleci script, which leverages the
>> _repeat job to help flesh this out.  With the merge to trunk it “should”
>> detect all new tests and run them, so its fair to expect this to be stable.
>>
>> I am not sure I understand this -
>>
>>
>> Sorry, I was trying to be explicit on the current state of test failures,
>> but was not clear.
>>
>> Accord adds a new system table to make state durable (similar to the
>> paxos table).  This added table causes a well known subset of python dtests
>> to fail as they are checking for the set of keyspaces/tables and detecting
>> a new one added, causing the test to fail.  We have been ignoring these
>> rather than changing python-dtest that way we pull in latest changes there
>> rather than forking and freezing the branch we use.  We will 100% update
>> python dtest along with the merge to trunk similar to all other
>> cross-cutting work done.  Trunk must not fail due to this table getting
>> added and must 100% be dealt with at the same time we merge.
>>
>> Do you plan to merge before fixing the tests? I am confused
>>
>>
>> Yes, its a blocker for trunk merge.
>>
>> I think Henrik brought an interesting point about releasable trunk. I
>> don’t think that we are yet there with the Jenkins issues, but we are
>> getting close and we strive for that, no?
>>
>>
>> As Josh calls out, we require releasable trunk as defined by Circle CI
>> (change was made in 4.1); my understanding is that people are working to
>> improve Jenkins, and once this work is done we can migrate off Circle CI.
>> Until that happens we need to make sure Circle CI is stable for every
>> commit.
>>
>> 3) what is the story with the ninja fixes?
>>
>>
>> Can’t speak for 100% of them, but from the ones I have seen it boils down
>> to “our merge process relies on humans to do the right thing, and people
>> mess up some times”, which sadly is true in trunk.  To avoid calling anyone
>> out other than myself, lets look at a specific one that I did
>>
>> $ git show f8243f41c9e96c4a0390558086ece078b0b6b15c
>> commit f8243f41c9e96c4a0390558086ece078b0b6b15c
>> Author: David Capwell 
>> Date:   Mon Jan 9 13:20:58 2023 -0800
>>
>> Ninja: Add AccordTestUtils.parse which was missing in the latest
>> commit
>>
>> diff --git
>> a/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java
>> b/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java
>> index 4adad32d8a..20142c439b 100644
>> --- a/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java
>> +++ b/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java
>> @@ -174,6 +174,14 @@ public class AccordTestUtils
>>  return statement.createTxn(ClientState.forInternalCalls(),
>> options);
>>  }
>>
>> +public static TransactionStatement parse(String query)
>> +{
>> +TransactionStatement.Parsed parsed =
>> (TransactionStatement.Parsed) QueryProcessor.parseStatement(query);
>> +Assert.assertNotNull(parsed);
>> +TransactionStatement statement = (TransactionStatement)
>> parsed.prepare(ClientState.forInternalCalls());
>> +return statement;
>> +}
>> +
>>  public static Txn createTxn(int readKey, int... writeKeys)
>>  {
>>
>>  StringBuilder sb = new StringBuilder("BEGIN TRANSACTION\n”);
>>
>> I merged but a specific method was missing, this caused the branch to
>> fail to compile so I ninja pa

[DISCUSS] cep-15-accord, cep-21-tcm, and trunk

[Caleb Rackliffe]

We've already talked a bit

about how and when the current Accord feature branch should merge to trunk.
Earlier today, the cep-21-tcm branch was created
 to house
ongoing work on Transactional Metadata.

Returning to CASSANDRA-18196
 (merging Accord to
trunk) after working on some other issues, I want to propose changing
direction slightly, and make sure this makes sense to everyone else.

1.) There are a few minor Accord test issues in progress that I'd like to
wrap up before doing anything, but those shouldn't take long. (See
CASSANDRA-18302  and
CASSANDRA-18320 .)
2.) Accord logically depends on Transactional Metadata.
3.) The new cep-21-tcm branch is going to have to be rebased to trunk on a
regular basis.

So...

4.) I would like to pause merging cep-15-accord to trunk, and instead
rebase it on cep-21-tcm until such time as the latter merges to trunk, at
which point cep-15-accord can be rebased to trunk again and then merged
when ready, nominally taking up the work of CASSANDRA-18196
 again.

Any objections to this?

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

[Caleb Rackliffe]

I actually did a dry run rebase of cep-15-accord on top of cep-21-tcm here:
https://github.com/apache/cassandra/pull/2227

It wasn't too terrible, and I was actually able to get the main CQL-based
Accord tests working as long as I disabled automatic forwarding of CAS and
SERIAL read operations to Accord. The bigger issue was test stability in
cep-21-tcm. I'm sure that will mature quickly here, and I created a few
issues to attach to the Transactional Metadata epic
<https://issues.apache.org/jira/browse/CASSANDRA-18330>.

In the meantime, I rebased cep-15-accord on trunk at
commit 3eb605b4db0fa6b1ab67b85724a9cfbf00aae7de. The option to finish the
remaining bits of CASSANDRA-18196
<https://issues.apache.org/jira/browse/CASSANDRA-18196> and merge w/o TCM
is still available, but it sounds like the question we want to answer is
whether or not we build a throwaway patch for linearizable epochs in lieu
of TCM?

FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration
risk while accepting the possible delivery risk.

On Fri, Mar 24, 2023 at 9:32 AM Josh McKenzie  wrote:

> making sure that joining and leaving nodes update some state via Paxos
> instead of via gossip
>
> What kind of a time delivery risk does coupling CEP-15 with CEP-21
> introduce (i.e. unk-unk on CEP-21 leading to delay cascades to CEP-15)?
> Seems like having a table we CAS state for on epochs wouldn't be *too 
> *challenging,
> but I'm not familiar w/the details so I could be completely off here.
>
> Being able to deliver both of these things on their own timetable seems
> like a pretty valuable thing assuming the lift required would be modest.
>
> On Fri, Mar 24, 2023, at 6:15 AM, Benedict wrote:
>
>
> Accord doesn’t have a hard dependency on CEP-21 fwiw, it just needs
> linearizable epochs. This could be achieved with a much more modest patch,
> essentially avoiding almost all of the insertion points of cep-21, just
> making sure that joining and leaving nodes update some state via Paxos
> instead of via gossip, and assign an epoch as part of the update.
>
> It would be preferable to use cep-21 since it introduces this
> functionality, and our intention is to use cep-21 for this. But it isn’t a
> hard dependency.
>
>
> On 22 Mar 2023, at 20:58, Henrik Ingo  wrote:
>
> 
> Since Accord depends on transactional meta-data... is there really any
> alternative than what you propose?
>
> Sure, if there is some subset of Accord that could be merged, while work
> continues on a branched that is based on CEP-21 branch, that would be
> great. Merging even a prototype of Accord to trunk probably has marketing
> value. (Don't laugh, many popular databases have had "atomic transactions,
> except if anyone executes DDL simultaneously".)
>
> On Tue, Mar 14, 2023 at 8:39 PM Caleb Rackliffe 
> wrote:
>
> We've already talked a bit
> <https://lists.apache.org/list?dev@cassandra.apache.org:2023-1:Merging%20CEP-15%20to%20trunk>
> about how and when the current Accord feature branch should merge to trunk.
> Earlier today, the cep-21-tcm branch was created
> <https://lists.apache.org/thread/qkwnhgq02cn12jon2h565kh2gpzp9rry> to
> house ongoing work on Transactional Metadata.
>
> Returning to CASSANDRA-18196
> <https://issues.apache.org/jira/browse/CASSANDRA-18196> (merging Accord
> to trunk) after working on some other issues, I want to propose changing
> direction slightly, and make sure this makes sense to everyone else.
>
> 1.) There are a few minor Accord test issues in progress that I'd like to
> wrap up before doing anything, but those shouldn't take long. (See
> CASSANDRA-18302 <https://issues.apache.org/jira/browse/CASSANDRA-18302>
>  and CASSANDRA-18320
> <https://issues.apache.org/jira/browse/CASSANDRA-18320>.)
> 2.) Accord logically depends on Transactional Metadata.
> 3.) The new cep-21-tcm branch is going to have to be rebased to trunk on
> a regular basis.
>
> So...
>
> 4.) I would like to pause merging cep-15-accord to trunk, and instead
> rebase it on cep-21-tcm until such time as the latter merges to trunk, at
> which point cep-15-accord can be rebased to trunk again and then merged
> when ready, nominally taking up the work of CASSANDRA-18196
> <https://issues.apache.org/jira/browse/CASSANDRA-18196> again.
>
> Any objections to this?
>
>
>
> --
>
>
>
> *Henrik Ingo*
>
> *c*. +358 40 569 7354
>
> *w*. *www.datastax.com <http://www.datastax.com>*
>
> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
> <https://www.linkedin.com/company/datastax/>
> <https://github.com/datastax/>
>
>
>

Re: [EXTERNAL] [DISCUSS] Next release date

[Caleb Rackliffe]

> I worry about the labor involved with having very large work like this
target a frozen branch and then also needing to pull it up to trunk. That
doesn't sound fun.

> I for one do not like to have release branches cut months before their
expected release.

> CEP-15 is mostly “net new stuff” and not “changes to existing stuff” from
my understanding?

Yes, yes, and yes. Agree w/ JD here. If we want CEP-21 to make it into 5.0,
I'd oppose freezing a 5.0 branch for QA until it merges.

SAI, Accord, UCS, and DDM are all features that can be pretty safely
disabled/not used, and unlike w/ CEP-21, we *should* be able to de-risk
those things much more easily before they merge.

To try to address Mick's question, assuming we have Accord depending on
TCM, I'm not sure how closely it can follow TCM in merging. We've been
talking about this In another thread: "[DISCUSS] cep-15-accord, cep-21-tcm,
and trunk", but trying to rebase cep-15-accord on cep-21-tcm is probably
the easiest way to minimize that window...

On Fri, Mar 24, 2023 at 10:41 AM Benjamin Lerer  wrote:

> I’m not sure what freezing early for “extra QA time” really buys us?
>
>
> Tries, SAI, UCS, extended TTLs, Java 17, Dynamic Data Masking... all those
> changes potentially introduce their set of issues/flakiness or other
> problems. The freeze will allow us to find those early and facilitate the
> integration of CEP 21 and 15 in my opinion.
>
> Le ven. 24 mars 2023 à 16:23, Jeremiah D Jordan 
> a écrit :
>
>> Given the fundamental change to how cluster operations work coming from
>> CEP-21, I’m not sure what freezing early for “extra QA time” really buys
>> us?  I wouldn’t trust any multi-node QA done pre commit.
>> What “stabilizing” do we expect to be doing during this time?  How much
>> of it do we just have to do again after those things merge?  I for one do
>> not like to have release branches cut months before their expected
>> release.  It just adds extra merge forward and “where should this go”
>> questions/overhead.  It could make sense to me to branch branch when CEP-21
>> merges and only let in CEP-15 after that.  CEP-15 is mostly “net new stuff”
>> and not “changes to existing stuff” from my understanding?  So no QA effort
>> wasted if it is done before it merges.
>>
>> -Jeremiah
>>
>> On Mar 24, 2023, at 9:38 AM, Josh McKenzie  wrote:
>>
>> I would like to propose a partial freeze of 5.0 in June
>>
>> My .02:
>> +1 to:
>> * partial freeze on an agreed upon date w/agreed upon other things that
>> can optionally go in after
>> * setting a hard limit on when we ship from that frozen branch regardless
>> of whether the features land or not
>>
>> -1 to:
>> * ever feature freezing trunk again. :)
>>
>> I worry about the labor involved with having very large work like this
>> target a frozen branch and then also needing to pull it up to trunk. That
>> doesn't sound fun.
>>
>> If we resurrected the discussion about cutting alpha snapshots from
>> trunk, would that change people's perspectives on the weight of this
>> current decision? We'd probably also have to re-open pandora's box talking
>> about the solidity of our API's on trunk as well if we positioned those
>> alphas as being stable enough to start prototyping and/or building future
>> applications against.
>>
>> On Fri, Mar 24, 2023, at 9:59 AM, Brandon Williams wrote:
>>
>> I am +1 on a 5.0 branch freeze.
>>
>> Kind Regards,
>> Brandon
>>
>> On Fri, Mar 24, 2023 at 8:54 AM Benjamin Lerer  wrote:
>> >>
>> >> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch?
>> >
>> >
>> > I was thinking of a cassandra-5.0 branch freeze. So branching 5.0 and
>> allowing only CEP-15 and 21 + bug fixes there.
>> > Le ven. 24 mars 2023 à 13:55, Paulo Motta  a
>> écrit :
>> >>
>> >> >  I would like to propose a partial freeze of 5.0 in June.
>> >>
>> >> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? I
>> agree with a branch freeze, but not with trunk freeze.
>> >>
>> >> I might work on small features after June and would be happy to delay
>> releasing these on 5.0+, but delaying merge to trunk until 5.0 is released
>> could be disruptive to contributors workflows and I would prefer to avoid
>> that if possible.
>> >>
>> >> On Fri, Mar 24, 2023 at 6:37 AM Mick Semb Wever 
>> wrote:
>> >>>
>> >>>
>>  I would like to propose a partial freeze of 5.0 in June.
>> 
>>  …
>> 
>>  This partial freeze will be valid for every new feature except
>> CEP-21 and CEP-15.
>> >>>
>> >>>
>> >>>
>> >>> +1
>> >>>
>> >>> Thanks for summarising the thread this way Benjamin. This addresses
>> my two main concerns: letting the branch/release date slip too much into
>> the unknown, squeezing GA QA efforts, while putting in place exceptional
>> waivers for CEP-21 and CEP-15.
>> >>>
>> >>> I hope that in the future we will be more willing to commit to the
>> release train model: less concerned about "what the next release contains";
>> more comfortable letting big features land where

Re: [EXTERNAL] [DISCUSS] Next release date

[Caleb Rackliffe]

> That does not mean that the code should not be stabilized before the
release. We had far less features in 4.1 and it took us 6 months to
release. Accepting more features until August will probably result in
extending the time needed to stabilize the release.

I think what I'm trying to get at is that CEP-21/TCM is almost certainly
the most invasive piece of work that will be in the release. This
probably means...

1.) "Stabilization" work for anything it touches before it merges will be
of limited value.
2.) No matter what else goes in before it merges, it will probably be the
bottleneck for stabilizing the release.

If we want to cut a release after TCM merges, let's cut a 5.0 branch,
stabilize it, and let SAI, Accord, et al. merge to trunk and appear in the
next release. If SAI or Accord are ready at roughly the same time, having
already been based on TCM in their feature branches and extremely
well-vetted (Harry, performance testing, simulator exposure), we can have
the discussion about merging them and then cutting a release branch.

On Fri, Mar 24, 2023 at 1:39 PM Benjamin Lerer  wrote:

> SAI, Accord, UCS, and DDM are all features that can be pretty safely
>> disabled/not used, and unlike w/ CEP-21, we *should* be able to de-risk
>> those things much more easily before they merge.
>
>
> That does not mean that the code should not be stabilized before the
> release. We had far less features in 4.1 and it took us 6 months to
> release. Accepting more features until August will probably result in
> extending the time needed to stabilize the release.
>
> I worry about the labor involved with having very large work like this
>> target a frozen branch and then also needing to pull it up to trunk. That
>> doesn't sound fun.
>>
> I am not sure I understand the issue of merging the work in a frozen
> branch. Should it not facilitate the work if the branch stops changing
> heavily? Regarding trunk, if most of us focus on stabilizing the 5.0 branch
> or on CEP-15 and 21 I do not think that it will change so much. Am I
> missing something?
>
> Le ven. 24 mars 2023 à 19:16, Caleb Rackliffe 
> a écrit :
>
>> > I worry about the labor involved with having very large work like this
>> target a frozen branch and then also needing to pull it up to trunk. That
>> doesn't sound fun.
>>
>> > I for one do not like to have release branches cut months before their
>> expected release.
>>
>> > CEP-15 is mostly “net new stuff” and not “changes to existing stuff”
>> from my understanding?
>>
>> Yes, yes, and yes. Agree w/ JD here. If we want CEP-21 to make it into
>> 5.0, I'd oppose freezing a 5.0 branch for QA until it merges.
>>
>> SAI, Accord, UCS, and DDM are all features that can be pretty safely
>> disabled/not used, and unlike w/ CEP-21, we *should* be able to de-risk
>> those things much more easily before they merge.
>>
>> To try to address Mick's question, assuming we have Accord depending on
>> TCM, I'm not sure how closely it can follow TCM in merging. We've been
>> talking about this In another thread: "[DISCUSS] cep-15-accord,
>> cep-21-tcm, and trunk", but trying to rebase cep-15-accord on cep-21-tcm
>> is probably the easiest way to minimize that window...
>>
>> On Fri, Mar 24, 2023 at 10:41 AM Benjamin Lerer 
>> wrote:
>>
>>> I’m not sure what freezing early for “extra QA time” really buys us?
>>>
>>>
>>> Tries, SAI, UCS, extended TTLs, Java 17, Dynamic Data Masking... all
>>> those changes potentially introduce their set of issues/flakiness or other
>>> problems. The freeze will allow us to find those early and facilitate the
>>> integration of CEP 21 and 15 in my opinion.
>>>
>>> Le ven. 24 mars 2023 à 16:23, Jeremiah D Jordan <
>>> jeremiah.jor...@gmail.com> a écrit :
>>>
>>>> Given the fundamental change to how cluster operations work coming from
>>>> CEP-21, I’m not sure what freezing early for “extra QA time” really buys
>>>> us?  I wouldn’t trust any multi-node QA done pre commit.
>>>> What “stabilizing” do we expect to be doing during this time?  How much
>>>> of it do we just have to do again after those things merge?  I for one do
>>>> not like to have release branches cut months before their expected
>>>> release.  It just adds extra merge forward and “where should this go”
>>>> questions/overhead.  It could make sense to me to branch branch when CEP-21
>>>> merges and only let in CEP-15 after that.  CEP-15 is mostly “net new stuff”
>>>> and not “changes to existing s

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

[Caleb Rackliffe]

To be clear, I’m not suggesting that the CEP-7 branch be rebased on cep-21-tcm rather than trunk.On Mar 24, 2023, at 2:12 PM, Josh McKenzie  wrote:FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk while accepting the possible delivery risk.What does the chain of rebases against trunk look like here? cep-21-tcm rebase, then cep-15 on cep-21-tcm, then cep-7 on cep-21-tcm, then a race on whichever of 15 or 7 merge after 21 goes into trunk? Or 7 based on 15, the other way around...I'm not actively working on any of these branches so take my perspective with that appropriate grain of salt, but the coupling of these things seems to have it's own kind of breed of integration pain to upkeep over time depending on how frequently we're rebasing against trunk.the question we want to answer is whether or not we build a throwaway patch for linearizable epochsDo we have an informed opinion on how long we think this would take? Seems like that'd help clarify whether or not there's contributors with the bandwidth and desire to even do that or whether everyone depending on cep-21 is our option.On Fri, Mar 24, 2023, at 1:30 PM, Caleb Rackliffe wrote:I actually did a dry run rebase of cep-15-accord on top of cep-21-tcm here: https://github.com/apache/cassandra/pull/2227It wasn't too terrible, and I was actually able to get the main CQL-based Accord tests working as long as I disabled automatic forwarding of CAS and SERIAL read operations to Accord. The bigger issue was test stability in cep-21-tcm. I'm sure that will mature quickly here, and I created a few issues to attach to the Transactional Metadata epic.In the meantime, I rebased cep-15-accord on trunk at commit 3eb605b4db0fa6b1ab67b85724a9cfbf00aae7de. The option to finish the remaining bits of CASSANDRA-18196 and merge w/o TCM is still available, but it sounds like the question we want to answer is whether or not we build a throwaway patch for linearizable epochs in lieu of TCM?FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk while accepting the possible delivery risk.On Fri, Mar 24, 2023 at 9:32 AM Josh McKenzie <jmcken...@apache.org> wrote:making sure that joining and leaving nodes update some state via Paxos instead of via gossipWhat kind of a time delivery risk does coupling CEP-15 with CEP-21 introduce (i.e. unk-unk on CEP-21 leading to delay cascades to CEP-15)? Seems like having a table we CAS state for on epochs wouldn't be too challenging, but I'm not familiar w/the details so I could be completely off here.Being able to deliver both of these things on their own timetable seems like a pretty valuable thing assuming the lift required would be modest.On Fri, Mar 24, 2023, at 6:15 AM, Benedict wrote:Accord doesn’t have a hard dependency on CEP-21 fwiw, it just needs linearizable epochs. This could be achieved with a much more modest patch, essentially avoiding almost all of the insertion points of cep-21, just making sure that joining and leaving nodes update some state via Paxos instead of via gossip, and assign an epoch as part of the update.It would be preferable to use cep-21 since it introduces this functionality, and our intention is to use cep-21 for this. But it isn’t a hard dependency.On 22 Mar 2023, at 20:58, Henrik Ingo <henrik.i...@datastax.com> wrote:Since Accord depends on transactional meta-data... is there really any alternative than what you propose?Sure, if there is some subset of Accord that could be merged, while work continues on a branched that is based on CEP-21 branch, that would be great. Merging even a prototype of Accord to trunk probably has marketing value. (Don't laugh, many popular databases have had "atomic transactions, except if anyone executes DDL simultaneously".)On Tue, Mar 14, 2023 at 8:39 PM Caleb Rackliffe <calebrackli...@gmail.com> wrote:We've already talked a bit about how and when the current Accord feature branch should merge to trunk. Earlier today, the cep-21-tcm branch was created to house ongoing work on Transactional Metadata.Returning to CASSANDRA-18196 (merging Accord to trunk) after working on some other issues, I want to propose changing direction slightly, and make sure this makes sense to everyone else.1.) There are a few minor Accord test issues in progress that I'd like to wrap up before doing anything, but those shouldn't take long. (See CASSANDRA-18302 and CASSANDRA-18320.)2.) Accord logically depends on Transactional Metadata.3.) The new cep-21-tcm branch is going to have to be rebased to trunk on a regular basis.So...4.) I would like to pause merging cep-15-accord to trunk, and instead rebase it on cep-21-tcm until such time as the latter merges to trunk, at which point cep-15-accord can be rebased to trunk again and then merged when ready, nominally taking up the work of CASSANDRA-18196 again.Any objections to this?--Henrik Ingoc. +358 40 569 7354 w. www.datastax.com      

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

[Caleb Rackliffe]

I agree there’s little point in litigating right now, given test stability (or lack thereof) in cep-21-tcm. Eventually, though, I’m more or less aligned w/ David in the sense that getting ourselves planted on top of TCM as soon as possible is a good idea.On Mar 24, 2023, at 3:04 PM, Benedict  wrote:It’s not even clear such an effort would need to be different from that used by cep-21. The point is that there’s not much point litigating this now when we can keep our options open and better decouple the projects, since I don’t think we lose much this way.On 24 Mar 2023, at 19:58, David Capwell  wrote:Assuming we do not release with it, then yes, as we wouldn’t need to maintain.  My point for this case was that I don’t feel the time cost is worth it, I am not -1 if someone wants to add, was more saying our time is better off else where.We currently don’t touch Transactional Metadata, we have custom logic (which relies on client to tell every instance in the cluster an update happened) that we are using right now in tests and deployed clusters.  So once we can integrate with Transactional Metadata, we stop relying on clients to tell us about topology changes… a custom/thrown away epoch generator would make this current process more user friendly, but would need to spend time to make sure stable when deployed on a clusterOn Mar 24, 2023, at 12:43 PM, Josh McKenzie  wrote:If this is in a release, we then need to maintain that feature, so would be against it.Isn't the argument that cep-21 provides this so we could just remove the temporary impl and point to the new facility for this generation?On Fri, Mar 24, 2023, at 3:22 PM, David Capwell wrote:the question we want to answer is whether or not we build a throwaway patch for linearizable epochsIf this is in a release, we then need to maintain that feature, so would be against it.If this is for testing, then I would argue the current world is “fine”… current world is hard to use and brittle (users need to tell accord that the cluster changed), but if accord is rebasing on txn metadata then this won’t be that way long (currently blocked from doing that due to txn metadata not passing all tests yet).On Mar 24, 2023, at 12:12 PM, Josh McKenzie  wrote:FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk while accepting the possible delivery risk.What does the chain of rebases against trunk look like here? cep-21-tcm rebase, then cep-15 on cep-21-tcm, then cep-7 on cep-21-tcm, then a race on whichever of 15 or 7 merge after 21 goes into trunk? Or 7 based on 15, the other way around...I'm not actively working on any of these branches so take my perspective with that appropriate grain of salt, but the coupling of these things seems to have it's own kind of breed of integration pain to upkeep over time depending on how frequently we're rebasing against trunk.the question we want to answer is whether or not we build a throwaway patch for linearizable epochsDo we have an informed opinion on how long we think this would take? Seems like that'd help clarify whether or not there's contributors with the bandwidth and desire to even do that or whether everyone depending on cep-21 is our option.On Fri, Mar 24, 2023, at 1:30 PM, Caleb Rackliffe wrote:I actually did a dry run rebase of cep-15-accord on top of cep-21-tcm here: https://github.com/apache/cassandra/pull/2227It wasn't too terrible, and I was actually able to get the main CQL-based Accord tests working as long as I disabled automatic forwarding of CAS and SERIAL read operations to Accord. The bigger issue was test stability in cep-21-tcm. I'm sure that will mature quickly here, and I created a few issues to attach to the Transactional Metadata epic.In the meantime, I rebased cep-15-accord on trunk at commit 3eb605b4db0fa6b1ab67b85724a9cfbf00aae7de. The option to finish the remaining bits of CASSANDRA-18196 and merge w/o TCM is still available, but it sounds like the question we want to answer is whether or not we build a throwaway patch for linearizable epochs in lieu of TCM?FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk while accepting the possible delivery risk.On Fri, Mar 24, 2023 at 9:32 AM Josh McKenzie <jmcken...@apache.org> wrote:making sure that joining and leaving nodes update some state via Paxos instead of via gossipWhat kind of a time delivery risk does coupling CEP-15 with CEP-21 introduce (i.e. unk-unk on CEP-21 leading to delay cascades to CEP-15)? Seems like having a table we CAS state for on epochs wouldn't be too challenging, but I'm not familiar w/the details so I could be completely off here.Being able to deliver both of these things on their own timetable seems like a pretty valuable thing assuming the lift required would be modest.On Fri, Mar 24, 2023, at 6:15 AM, Benedict wrote:Accord doesn’t have a hard dependency on CEP-21 fwiw, it just needs linearizable epochs. This cou

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

[Caleb Rackliffe]

Minimizing uncertainty is a nice abstract goal. What I worry about is that we ultimately create more of it (and more work/thrashing for ourselves) by not basing Accord on TCM at the earliest responsible moment.Again, although I created this thread, the state of the world is telling me a decision doesn’t need to be made quite yet.On Mar 27, 2023, at 5:29 AM, Benedict  wrote:I don’t know that TCM is a greater source of uncertainty. There is a degree of uncertainty about that :)I just think it’s better not to compound uncertainties, at least while it is not costly to avoid it.On 27 Mar 2023, at 08:27, Henrik Ingo  wrote:Seems like this thread is the more appropriate one for Accord/TCM discussionIMO the priority here should be:1. Release CEP-15 as part of 5.0, this year, with or without  CEP-21.2. Minimize work arising from porting between branches. (e.g. first onto CEP-21, then to trunk, or vice versa. But also then between 5.0 and trunk)3. Minimize work arising from temporary solutions that support goal #1If CEP-21 is the major source of uncertainty right now, then we should merge CEP-15 to trunk independently. If CEP-15 depending on CEP-21 just means an additional month (or two) delay, then that's fine and there's no need to do additional work just to get some preview release out earlier.henrikOn Sat, Mar 25, 2023 at 4:17 AM Caleb Rackliffe <calebrackli...@gmail.com> wrote:I agree there’s little point in litigating right now, given test stability (or lack thereof) in cep-21-tcm. Eventually, though, I’m more or less aligned w/ David in the sense that getting ourselves planted on top of TCM as soon as possible is a good idea.On Mar 24, 2023, at 3:04 PM, Benedict <bened...@apache.org> wrote:It’s not even clear such an effort would need to be different from that used by cep-21. The point is that there’s not much point litigating this now when we can keep our options open and better decouple the projects, since I don’t think we lose much this way.On 24 Mar 2023, at 19:58, David Capwell <dcapw...@apple.com> wrote:Assuming we do not release with it, then yes, as we wouldn’t need to maintain.  My point for this case was that I don’t feel the time cost is worth it, I am not -1 if someone wants to add, was more saying our time is better off else where.We currently don’t touch Transactional Metadata, we have custom logic (which relies on client to tell every instance in the cluster an update happened) that we are using right now in tests and deployed clusters.  So once we can integrate with Transactional Metadata, we stop relying on clients to tell us about topology changes… a custom/thrown away epoch generator would make this current process more user friendly, but would need to spend time to make sure stable when deployed on a clusterOn Mar 24, 2023, at 12:43 PM, Josh McKenzie <jmcken...@apache.org> wrote:If this is in a release, we then need to maintain that feature, so would be against it.Isn't the argument that cep-21 provides this so we could just remove the temporary impl and point to the new facility for this generation?On Fri, Mar 24, 2023, at 3:22 PM, David Capwell wrote:the question we want to answer is whether or not we build a throwaway patch for linearizable epochsIf this is in a release, we then need to maintain that feature, so would be against it.If this is for testing, then I would argue the current world is “fine”… current world is hard to use and brittle (users need to tell accord that the cluster changed), but if accord is rebasing on txn metadata then this won’t be that way long (currently blocked from doing that due to txn metadata not passing all tests yet).On Mar 24, 2023, at 12:12 PM, Josh McKenzie <jmcken...@apache.org> wrote:FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk while accepting the possible delivery risk.What does the chain of rebases against trunk look like here? cep-21-tcm rebase, then cep-15 on cep-21-tcm, then cep-7 on cep-21-tcm, then a race on whichever of 15 or 7 merge after 21 goes into trunk? Or 7 based on 15, the other way around...I'm not actively working on any of these branches so take my perspective with that appropriate grain of salt, but the coupling of these things seems to have it's own kind of breed of integration pain to upkeep over time depending on how frequently we're rebasing against trunk.the question we want to answer is whether or not we build a throwaway patch for linearizable epochsDo we have an informed opinion on how long we think this would take? Seems like that'd help clarify whether or not there's contributors with the bandwidth and desire to even do that or whether everyone depending on cep-21 is our option.On Fri, Mar 24, 2023, at 1:30 PM, Caleb Rackliffe wrote:I actually did a dry run rebase of cep-15-accord on top of cep-21-tcm here: https://github.com/apache/cassandra/pull/2227It wasn't too terrible, and I was actually abl

Re: [DISCUSS] Introduce DATABASE as an alternative to KEYSPACE

[Caleb Rackliffe]

KEYSPACE isn’t a terrible name for a namespace that also configures how keys are replicated. NAMESPACE is accurate but not comprehensive. DATABASE doesn’t seem to have the advantages of either.I’m neutral on NAMESPACE and slightly -1 on DATABASE. It’s hard for me to believe KEYSPACE is really a stumbling block for new users, especially when it connotes something those users should understand about them (the replication configuration).On Apr 5, 2023, at 4:16 AM, Aleksey Yeshchenko  wrote:FYI we support SCHEMA as an alias to KEYSPACE today (have since always). Can use CREATE SCHEMA in place of CREATE KEYSPACE, etc. On 4 Apr 2023, at 19:23, Henrik Ingo  wrote:I find the Postgres terminology overly complex. Where most SQL databases will have several *databases*, each containing several *tables*, in Postgres we have namespaces, databases, schemas and tables...Oracle seems to also use the words database, schema and tables. I don't know if it has namespaces.Ah, ok, so SQL Server actually is like Oracle too!So in MySQL, referring unambiguously (aka full path) to a table would be:    SELECT * FROM mydb.mytable;Whereas in Postgresql and Oracle and SQL Server you'd have to:    SELECT * FROM mydb.myschema.mytable;   /* And I don't even know what to do with the namespace! */https://www.postgresql.org/docs/current/catalog-pg-namespace.htmlhttps://www.postgresql.org/docs/current/ddl-schemas.htmlhttps://docs.oracle.com/database/121/ADMQS/GUID-6E0CE8C9-7DC4-450C-BAE0-2E1CDD882993.htm#ADMQS0821https://docs.oracle.com/database/121/ADMQS/GUID-8AC1A325-3542-48A0-9B0E-180D633A5BD1.htm#ADMQS081https://learn.microsoft.com/en-us/sql/t-sql/statements/create-schema-transact-sql?view=sql-server-ver16https://learn.microsoft.com/en-us/sql/t-sql/statements/create-database-transact-sql?view=sql-server-ver16&tabs=sqlpoolThe Microsoft docs perhaps best explain the role of each: The Database contains the configuration of physical things like where on disk is the database stored. The Schema on the other hand contains "logical" objects like tables, views andprocedures.MongoDB has databases and collections. As an easter egg / inside joke, it also supports the command `SHOW TABLES` as a synonym for collections. A TABLESPACE btw is something else completely: https://docs.oracle.com/database/121/ADMQS/GUID-F05EE514-FFC6-4E86-A592-802BA5A49254.htm#ADMQS12053Personally I would be in favor of introducing `DATABASE` as a synonym for KEYSPACE. The latter could remain the "official" usage.henrikOn Tue, Apr 4, 2023 at 8:32 PM Jacek Lewandowski  wrote:While for someone who already knows Cassandra keyspace is something natural, for newcomers it is yet another concept to understand. If namespace is used in PostgreSQL, it sounds even better to me.Thanks,- - -- --- -  -Jacek Lewandowskiwt., 4 kwi 2023 o 19:07 Rahul Xavier Singh  napisał(a):My 2 cents: Keeping it keyspace works for me, namespace could be cool also since we decide where that namespace exists in relation to Datacenters, etc.  In our case, a Keyspace is more similar to a namespace than it is to a database since we expect all the UDTs,/UDFs, indexes to refer to only the tables in that keyspace/namespace. Alternatively interesting to observe and throw some fuel into the discussion , looking at the Postgres (only because there are many distributed databases that are now PG compliant) :From the interwebs: In PostgreSQL, a schema is a namespace that contains named database objects such as tables, views, indexes, data types, functions, stored procedures and operators. A database can contain one or multiple schemas and each schema belongs to only one database.I used to gripe about this but as a platform gets more complex it is useful to organize PG DBs into schemas. In C* world, I found myself doing similar things by having a prefix : e.g. appprefix_system1 appprefix_system2 ...Rahul SinghChief Executive Officer | Business Platform Architect
m: 202.905.2818 e: rahul.si...@anant.us li: http://linkedin.com/in/xingh ca: http://calendly.com/xinghWe create, support, and manage real-time global data & analytics platforms for the modern enterprise.Anant | https://anant.us3 Washington Circle, Suite 301Washington, D.C. 20037http://Cassandra.Link : The best resources for Apache CassandraOn Tue, Apr 4, 2023 at 12:52 PM Jeff Jirsa  wrote:KEYSPACE at least makes sense in the context that it is the unit that defines how those partitions keys are going to be treated/replicatedDATABASE may be ambiguous, but it's ambiguity shared across the industry. Creating a new name like TABLESPACE or TABLEGROUP sounds horrible because it'll be unique to us in the world, and therefore unintuitive for new users.On Tue, Apr 4, 2023 at 9:36 AM Josh McKenzie  wrote:I think there's competing dynamics here.1) KEYSPACE isn't that great of a name; it's not a space in which keys are necessarily unique, and you can't ad

Re: [DISCUSS] CEP-29 CQL NOT Operator

[Caleb Rackliffe]

+1 to the proposal from a CQL perspective

*However*, whether we do this in the context of simple partition
restriction, a global index query, or a partition-restricted index query,
the NOT operator is most likely to be useful only in a post-filtering
capacity. (ex. WHERE indexed_set CONTAINS { 'foo'} AND indexed_set NOT
CONTAINS { 'bar' })

Using Lucene as an example, you might remember that it doesn't (at least
IIRC) allow single predicate NOT queries. (See
https://stackoverflow.com/questions/3604771/not-query-in-lucene) It's easy
for an inverted index to find matches efficiently, but not so easy for it
to find non-matches. This is similar to, but even less-straightforward
than, the issue you have w/ boolean queries when you query the less
selective of the two possible values. You can create an accompanying
"negated" index, but that's not free, of course.

Again, not necessarily a problem w/ the CEP, but want to call out the
potential complication...

On Thu, Apr 6, 2023 at 4:01 PM Jeremy Hanna 
wrote:

> Considering all of the examples require using ALLOW FILTERING with the
> partition key specified, I think it's appropriate to consider separating
> out use of ALLOW FILTERING within a partition versus ALLOW FILTERING across
> the whole table.  A few years back we had a discussion about this in ASF
> slack in the context of capability restrictions and it seems relevant
> here.  That is, we don't want people to get comfortable using ALLOW
> FILTERING across the whole table.  However, there are times when ALLOW
> FILTERING within a partition is reasonable.
>
> Ticket to discuss separating them out:
> https://issues.apache.org/jira/browse/CASSANDRA-15803
> Summary: Perhaps add an optional [WITHIN PARTITION] or something similar
> to make it backwards compatible and indicate that this is purely within the
> specified partition.
>
> This also gives us the ability to disallow table scan types of ALLOW
> FILTERING from a guard rail perspective, because the intent is explicit.
> That operators could disallow ALLOW FILTERING but allow ALLOW FILTERING
> WITHIN PARTITION, or whatever is decided.
>
> I do NOT want to hijack a good discussion but I thought this separation
> could be useful within this context.
>
> Jeremy
>
> On Apr 6, 2023, at 3:00 PM, Patrick McFadin  wrote:
>
> I love that this is finally coming to Cassandra. Absolutely hate that,
> once again, we'll be endorsing the use of ALLOW FILTERING. This is an
> anti-pattern that keeps getting legitimized.
>
> Hot take: Should we just not do Milestones 1 and 2 and wait for an
> index-only Milestone 3?
>
> Patrick
>
> On Thu, Apr 6, 2023 at 10:04 AM David Capwell  wrote:
>
>> Overall I welcome this feature, was trying to use this around 1-2 months
>> back and found we didn’t support, so glad to see it coming!
>>
>> From a testing point of view, I think we would want to have good fuzz
>> testing covering complex types (frozen/non-frozen collections, tuples, udt,
>> etc.), and reverse ordering; both sections tend to cause the most problem
>> for new features (and existing ones)
>>
>> We also will want a way to disable this feature, and optionally disable
>> at different sections (such as m2’s NOT IN for partition keys).
>>
>> > On Apr 4, 2023, at 2:28 AM, Piotr Kołaczkowski 
>> wrote:
>> >
>> > Hi everyone!
>> >
>> > I created a new CEP for adding NOT support to the query language and
>> > want to start discussion around it:
>> >
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-29%3A+CQL+NOT+operator
>> >
>> > Happy to get your feedback.
>> > --
>> > Piotr
>>
>>
>

Re: [DISCUSS] Next release date

[Caleb Rackliffe]

> Once all CEPs except CEP-21 and CEP-15 land we branch cassandra-5.0

For the record, I'm not convinced this is necessarily better than just
cutting a cassandra-5.0 branch on 1 October.

On Mon, Apr 17, 2023 at 2:30 AM Mick Semb Wever  wrote:

> 2. When CEP-15 lands we cut alpha1,
>> 2a. The deadline is first week of October, anything not yet in
>> cassandra-5.0 is not in 5.0,
>> 2b. We expect a minimum two months of testing and beta+rc releases
>> to get to GA.
>>
>> To clarify, is the intent here to say "The deadline for cutoff is 1st
>> week of October for everything, including CEP-15"? Or is the intent to say
>> "we don't cut alpha1 until CEP-15 lands"?
>>
>
>
> The former. The first week of October will be the full feature freeze on
> the cassandra-5.0 branch.
>
>
>

Re: [DISCUSS] Next release date

[Caleb Rackliffe]

...or just cutting a 5.0 branch when CEP-21 is ready.

There's nothing stopping us from testing JDK17 and TTL bits in trunk before
that.

On Mon, Apr 17, 2023 at 11:25 AM Caleb Rackliffe 
wrote:

> > Once all CEPs except CEP-21 and CEP-15 land we branch cassandra-5.0
>
> For the record, I'm not convinced this is necessarily better than just
> cutting a cassandra-5.0 branch on 1 October.
>
> On Mon, Apr 17, 2023 at 2:30 AM Mick Semb Wever  wrote:
>
>> 2. When CEP-15 lands we cut alpha1,
>>> 2a. The deadline is first week of October, anything not yet in
>>> cassandra-5.0 is not in 5.0,
>>> 2b. We expect a minimum two months of testing and beta+rc releases
>>> to get to GA.
>>>
>>> To clarify, is the intent here to say "The deadline for cutoff is 1st
>>> week of October for everything, including CEP-15"? Or is the intent to say
>>> "we don't cut alpha1 until CEP-15 lands"?
>>>
>>
>>
>> The former. The first week of October will be the full feature freeze on
>> the cassandra-5.0 branch.
>>
>>
>>

Re: [DISCUSS] Next release date

[Caleb Rackliffe]

What I'm trying to propose is that we either...

a.) Pick the latest responsible (preserves our desired timeline for GA)
date to cut a 5.0 branch, and not let anything else in after, including
CEP-15/CEP-21.
b.) Cut a 5.0 branch when the major release-defining element (maybe
CEP-21?) merges to trunk, with the shared understanding (possibly what
we're disagreeing about) that very little of what we need to test/de-risk
is going to be inhibited by not cutting that branch earlier (and that
certain testing efforts would be almost wholesale wasted if done
beforehand).

i.e. Have one freeze/branch point. Pick a date and don't make exceptions,
or pick the feature and *potentially* adjust the date.

On Mon, Apr 17, 2023 at 2:07 PM Josh McKenzie  wrote:

> So to bring us back to the goals and alignment here:
>
> With the following intentions:
> - moving towards the goal of annual releases, with a cadence 12±3 months
> apart,
> - the branch to GA period being 2-3 months,
> - avoiding any type of freeze on trunk,
> - getting a release out by December's Summit,
> - freeing up folk to start QA (that makes sense to start) immediately
>
> So what I *think* falls out logically:
>
> 1. We branch cassandra-5.0 on August 1st
> 2. We expect an 8-12 week validation cycle which means GA Oct1-Nov1.
> 3a. If we allow merge of CEP-15 / CEP-21 after branch, we risk
> invalidating stabilization and risk our 2023 GA date
> 3b. If we don't allow merge of CEP-15 / CEP-21 after branch, we risk
> needing a fast-follow release and don't have functional precedent for the
> snapshots we earlier agreed upon doing.
>
> Does that distill it and match everyone else's understanding?
>
> On Mon, Apr 17, 2023, at 2:20 PM, Mick Semb Wever wrote:
>
>
>
> On Mon, 17 Apr 2023 at 19:31, Caleb Rackliffe 
> wrote:
>
> ...or just cutting a 5.0 branch when CEP-21 is ready.
>
> There's nothing stopping us from testing JDK17 and TTL bits in trunk
> before that.
>
> On Mon, Apr 17, 2023 at 11:25 AM Caleb Rackliffe 
> wrote:
>
> > Once all CEPs except CEP-21 and CEP-15 land we branch cassandra-5.0
>
> For the record, I'm not convinced this is necessarily better than just
> cutting a cassandra-5.0 branch on 1 October.
>
>
>
> How else would this work without being akin to a feature freeze on trunk.
>
> We want (need) as much time as possible to test. We have no evidence that
> it will be quicker than 4.1, we have to create that evidence. Those folk
> that free up and are ready to get ahead and de-risk our testing efforts
> should be given a release branch to make their work easier and to give us
> that evidence in a more controlled manner so that we can plan better next
> time. Appreciate that there's one too many variables here, but I'm sticking
> up for the testing efforts here.
>
>
>

Re: [DISCUSS] Next release date

[Caleb Rackliffe]

> My personal .02: I think we should consider branching 5.0 September 1st.
That gives us basically 12 weeks for folks to do their testing and for us
to stabilize anything that's flaky in circle or regressed in ASF CI.

WFM, if that means we branch there and anything not already merged has to
wait


On Mon, Apr 17, 2023 at 3:37 PM Josh McKenzie  wrote:

> it's (b) for me, and everything minus 21 and 15 is defining enough to
> warrant the branching and a checkpoint where testing can start
>
> Ok, I don't follow.
>
> There's three different ways I can read what you're saying here:
>
>1. "Everything we have targeting 5.x is substantial and we can branch
>when it's done", that'd mean 600+ open tickets, so it can't be that:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20cassandra%20and%20fixversion%20%3D%205.x%20and%20resolution%20%3D%20unresolved
>2. "Everything we've *already merged today* targeting 5.x is
>substantial and we can branch now"... I don't think that's quite right?
>Since that'd put the release far too early after December '22.
>3. "Everything we expect to merge by August 1st, regardless of CEP-21
>and CEP-15, is substantial enough for us to cut a release then" - that's
>arguing for a feature-driven release rather than a train right?
>
> Sorry; I'm definitely not *trying* to be obtuse, I'm just having a hard
> time understanding how what the two of you are saying actually lines up.
>
> My personal .02: I think we should consider branching 5.0 September 1st.
> That gives us basically 12 weeks for folks to do their testing and for us
> to stabilize anything that's flaky in circle or regressed in ASF CI.
>
> If CEP-15 or CEP-21 land shortly after (early September), we can cross
> that bridge when the time comes.
>
> That's my hot take. We move to a train model and stick with it, and we
> start to get comfortable with cutting feature previews or snapshot alphas
> like we agreed to for earlier access to new stuff.
>
> On Mon, Apr 17, 2023, at 4:25 PM, Mick Semb Wever wrote:
>
>
> b.) Cut a 5.0 branch when the major release-defining element (maybe
> CEP-21?) merges to trunk, with the shared understanding (possibly what
> we're disagreeing about) that very little of what we need to test/de-risk
> is going to be inhibited by not cutting that branch earlier (and that
> certain testing efforts would be almost wholesale wasted if done
> beforehand).
>
>
>
> Yup, it's (b) for me, and everything minus 21 and 15 is defining enough to
> warrant the branching and a checkpoint where testing can start and not be
> wasted.  I understand that cep-21 changes a lot and that impacts testing,
> but I wholeheartedly trust testers to be taking this into consideration.
>
>
>

Re: [DISCUSS] Next release date

[Caleb Rackliffe]

> Caleb, you appear to be the only one objecting, and it does not appear
that you have made any compromises in this thread.

All I'm really objecting to is making special exceptions for particular
CEPs in relation to our freeze date. In other words, let's not have a
pseudo-freeze date and a "real" freeze date, when the thing that makes the
latter supposedly necessary is a very invasive change to the database that
risks our desired GA date. Also, again, I don't understand how cutting a
5.0 branch makes anything substantially easier to start testing. Perhaps
I'm the only one who thinks this. If so, I'm not going to make further
noise about it.

On Tue, Apr 18, 2023 at 7:26 AM Henrik Ingo 
wrote:

> I forgot one last night:
>
> From Benjamin we have a question that I think went unanswered?
>
> *> Should it not facilitate the work if the branch stops changing heavily?*
>
> This is IMO a good perspective. To me it seems weird to be too hung up on
> a "hard limit" on a specific day, when we are talking about merges where a
> single merge / rebase takes more than one day. We will have to stop merging
> smaller work to trunk anyway, when CEP-21 is being merged. No?
>
> henrik
>
> On Tue, Apr 18, 2023 at 3:24 AM Henrik Ingo 
> wrote:
>
>> Trying to collect a few loose ends from across this thread
>>
>> *> I'm receptive to another definition of "stabilize", *
>>
>> I think the stabilization period implies more than just CI, which is
>> mostly a function of unit tests working correctly. For example, at Datastax
>> we have run a "large scale" test with >100 nodes, over several weeks, both
>> for 4.0 and 4.1. For obvious reasons such tests can't run in nightly CI
>> builds.
>>
>> Also it is not unusual that during the testing phase developers or
>> specialized QA engineers can develop new tests (which are possibly added to
>> CI) to improve coverage for and especially targeting new features in the
>> release. For example the fixes to Paxos v2 were found by such work before
>> 4.1.
>>
>> Finally, maybe it's a special case relevant only for  this release, but
>> as a significant part of the Datastax team has been focused on porting
>> these large existing features from DSE, and to get them merged before the
>> original May date, we also have tens of bug fixes waiting to be upstreamed
>> too. (It used to be an even 100, but I'm unsure what the count is today.)
>>
>> In fact! If you are worried about how to occupy yourself between a May
>> "soft freeze" and September'ish hard freeze, you are welcome to chug on
>> that backlog. The bug fixes are already public and ASL licensed, in the 4.0
>> based branch here
>> .
>> Failed with an unknown error.
>>
>> *> 3a. If we allow merge of CEP-15 / CEP-21 after branch, we risk
>> invalidating stabilization and risk our 2023 GA date*
>>
>> I think this is the assumption that I personally disagree with. If this
>> is true, why do we even bother running any CI before the CEP-21 merge? It
>> will all be invalidated anyway, right?
>>
>> In my experience, it is beneficial to test as early as possible, and at
>> different checkpoints during development. If we wouldn't  do it, and we
>> find some issue in late November, then the window to search for the commit
>> that introduced the regression is all the way back to the 4.1 GA. If on the
>> other hand the same test was already rune during the soft freeze, then we
>> can know that we may focus our search onto CEP-15 and CEP-21.
>>
>>
>> *> get comfortable with cutting feature previews or snapshot alphas like
>> we agreed to for earlier access to new stuff*
>>
>> Snapshots are in fact a valid compromise proposal: A snapshot would
>> provide a constant version / point in time to focus testing on, but on the
>> other hand would allow trunk (or the 5.0 branch, in other proposals) to
>> remain open to new commits. Somewhat "invalidating" the testing work, but
>> presumably the branch will be relatively calm anyway. Which leads me to 2
>> important questions:
>>
>> *WHO would be actively merging things into 5.0 during June-August? *
>>
>> By my count at that point I expect most contributors to either furiously
>> work on Acccord and TCM, or work on stabilization (tests, fixes).
>>
>> Also, if someone did contribute new feature code during this time, they
>> might find it hard to get priority for reviews, if others are focused on
>> the above tasks.
>>
>> Finally, I expect most Europeans to be on vacation 33% of that time.
>> Non-Europeans may want to try it too!
>>
>>
>> *WHAT do we expect to get merged during June-August?*
>>
>> Compared to the tens of thousands of lines of code being merged by
>> Accord, SAI, UCS and Tries... I imagine even the worst case during a
>> non-freeze in June-August would be just a tiny percentage of the large CEPs.
>>
>> In this thread I only see Paulo announcing an intent to commit against
>> trunk during a soft freeze, and even he agrees with a 5.0 branch freeze

Re: [DISCUSS] New data type for vector search

[Caleb Rackliffe]

I don’t have a lot to add here, other than to say I’m broadly in agreement w/ David on syntax preference, element selectability, and making this a new type that roughly corresponds to a primitive (non-null-allowing) array.On Apr 27, 2023, at 9:18 PM, Anthony Grasso  wrote:It would be strange for this declaration to look different from other collection types. We may want to reconsider using the collection syntax. I also like the idea of the vector dimensions being declared with the VECTOR keyword. An alternative syntax option to explore is:VECTOR[size]On Fri, 28 Apr 2023 at 10:49, Josh McKenzie  wrote:From a machine learning perspective, vectors are a well-known concept that are effectively immutable fixed-length n-dimensional values that are then later used either as part of a model or in conjunction with a model after the fact.While we could have this be non-frozen and not call it a vector, I'd be inclined to still make the argument for a layer of syntactic sugar on top that met ML users where they were with concepts they understood rather than forcing them through the cognitive lift of figuring out the Cassandra specific contortions to replicate something that's ubiquitous in their space. We did the same "Cassandra-first" approach with our JSON support and that didn't do us any favors in terms of adoption and usage as far as I know.So is the goal here to provide something specific and idiomatic for the ML community or is the goal to make a primitive that's C*-centric that then another layer can write to? I personally argue for the former; I don't see this specific data type going away any time soon.On Thu, Apr 27, 2023, at 12:39 PM, David Capwell wrote:but as you point out it has the problem of allowing nulls.If nulls are not allowed for the elements, then either we need  a) a new type, or b) add some way to say elements may not be null…. As much as I do like b, I am leaning towards new type for this use case.So, to flesh out the type requirements I have seen so far1) represents a fixed size array of element type* on write path we will need to validate this2) element may not be null* on write path we will need to validate this3) “frozen” (is this really a requirement for the type or is this just simpler for the ANN work?  I feel that this shouldn’t be a requirement)4) works for all types (my requirement; original proposal is float only, but could logically expand to primitive types)Anything else?The key thing about a vector is that unlike lists or tuples you really don't care about individual elements, you care about doing vector and matrix multiplications with the thing as a unit. That maybe true for this use case, but “should” this be true for the type itself?  I feel like no… if a user wants the Nth element of a vector why would we block them?  I am not saying the first patch, or even 5.0 adds support for index access, I am just trying to push back saying that the type should not block this.(Maybe this is making the case for VECTOR FLOAT[N] rather than FLOAT VECTOR[N].)Now that nulls are not allowed, I have mixed feelings about FLOAT[N], I prefer this syntax but that limitation may not be desired for all use cases… we could always add LIST and ARRAY later to address that case.In terms of syntax I have seen, here is my ordered preference:1) TYPE[size] - have mixed feelings due to non-null, but still prefer it2) QUALIFIER TYPE[size] - QUALIFIER is just a Term we use to denote this semantic…. Could even be NON NULL TYPE[size]On Apr 27, 2023, at 9:00 AM, Benedict  wrote:That’s a bounded ring buffer, not a fixed length array.This definitely isn’t a tuple because the types are all the same, which is pretty crucial for matrix operations. Matrix libraries generally work on arrays of known dimensionality, or sparse representations.Whether we draw any semantic link between the frozen list and whatever we do here, it is fundamentally a frozen list with a restriction on its size. What we’re defining here are “statically” sized arrays, whereas a frozen list is essentially a dynamically sized array.I do not think vector is a good name because vector is used in some other popular languages to mean a (dynamic) list, which is confusing when we also have a list concept.I’m fine with just using the FLOAT[N] syntax, and drawing no direct link with list. Though it is a bit strange that this particular type declaration looks so different to other collection types.On 27 Apr 2023, at 16:48, Jeff Jirsa  wrote:On Thu, Apr 27, 2023 at 7:39 AM Jonathan Ellis  wrote:It's been a while, so I may be missing something, but do we already have fixed-size lists?  If not, I don't see why we'd try to make this fit into a List-shaped problem.We do not. The proposal got closed as wont-fix  https://issues.apache.org/jira/browse/CASSANDRA-9110

Re: [POLL] Vector type for ML

[Caleb Rackliffe]

Did we agree on a CQL syntax?

On Wed, May 3, 2023 at 2:06 PM Rahul Xavier Singh <
rahul.xavier.si...@gmail.com> wrote:

> I like this approach. Thank you for those working on this vector search
> initiative.
>
> Here's the feedback from my "user" hat for someone who is looking at
> databases / indexes for my next LLM app.
>
> Can I take some python code and go from using an in memory vector store
> like numpy or FAISS to something else? How easy is it for me to take my
> python code and get it to work with this new external service which is no
> longer just a library?
> There's also tons of services that I can run on docker e.g. milvus,
> redissearch, typesense, elasticsearch, opensearch and I may hit a hurdle
> when trying to do a lot more data, so I look at Cassandra Vector Search.
> Because I am familiar with SQL , Cassandra looks appealing since I can
> potentially use "cql_agent" lib ( to be created for langchain and we're
> looking into that now) or an existing CassandraVectorStore class?
>
> In most of these scenarios, if people are using langchain, llamaindex, the
> underlying implementation is not as important since we shield the user from
> CQL data types except at schema creation and most of this libs can be
> opinionated and just suggest a generic schema.
>
> The ideal world is if I can just dump text into a field and do a natural
> language query on it and have my DB do the embeddings for the document, and
> then for the query for me. For now the libs can manage all that and they do
> that well. We just need the interface to stay consistent and be relatively
> easy to query in CQL. The most popular index in LLM retrieval augmented
> patterns is pinecone. You make an index, you upsert, and then you query.
> It's not assumed that you are also giving it content, though you can send
> it metadata to have the document there.
>
> If we can have a similar workflow e.g. create a table with a vector type
> OR create a table with an existing type and then add an index to it, no one
> is going to sleep over it as long as it works. Having the ability to take a
> table that has data, and then add a vector index doesn't make it any
> different than adding a new field since I've got to calculate the
> embeddings anyways.
>
> Would love to see how the CQL ends up looking like.
> Rahul Singh
>
> Chief Executive Officer | Business Platform Architect m: 202.905.2818 e:
> rahul.si...@anant.us li: http://linkedin.com/in/xingh ca:
> http://calendly.com/xingh
>
> *We create, support, and manage real-time global data & analytics
> platforms for the modern enterprise.*
>
> *Anant | https://anant.us *
>
> 3 Washington Circle, Suite 301
>
> Washington, D.C. 20037
>
> *http://Cassandra.Link * : The best resources for
> Apache Cassandra
>
>
> On Tue, May 2, 2023 at 6:39 PM Patrick McFadin  wrote:
>
>> \o/
>>
>> Bring it in team. Group hug.
>>
>> Now if you'll excuse me, I'm going to go build my preso on how Cassandra
>> is the only distributed database you can do vector search in an ACID
>> transaction.
>>
>> Patrick
>>
>> On Tue, May 2, 2023 at 3:27 PM Jonathan Ellis  wrote:
>>
>>> I had a call with David.  We agreed that we want a "vector" data type
>>> with these properties
>>>
>>> - Fixed length
>>> - No nulls
>>> - Random access not supported
>>>
>>> Where we disagreed was on my proposal to restrict vectors to only
>>> numeric data.  David's points were that
>>>
>>> (1) He has a use case today for a data type with the other vector
>>> properties,
>>> (2) It doesn't seem reasonable to create two data types with the same
>>> properties, one of which is restricted to numerics, and
>>> (3) The restrictions that I want for numeric vectors make more sense at
>>> the index and function level, than at the type level.
>>>
>>> I'm ready to concede that David has the better case here and move
>>> forward with a vector implementation without that restriction.
>>>
>>> On Tue, May 2, 2023 at 4:03 PM David Capwell  wrote:
>>>
  How about it, David? Did you already make this?


 I checked out the patch, fixed serialize/deserialize, added the
 constraints, then added a composeForFloat(ByteBuffer), with this the impact
 to the POC patch was the following

 1) move away from VectorType.instance.serializer().deserialize(bb) to
 type.composeForFloat(bb), both return float[]
 2) change the index validate logic to move away from checking
 VectorType and instead check for that plus the element type == FloatType.
 I didn’t bother to do this as its trivial

 David. End this argument. SHOW THE CODE!


 If this argument ends and people are cool with vector supporting
 abstract type, more than glad to help get this in.

 On May 2, 2023, at 1:53 PM, Jeremy Hanna 
 wrote:

 I'm all for bringing more functionality to the masses sooner, but the
 original idea has a very very specific use case.  Do we have use ca

Re: [POLL] Vector type for ML

[Caleb Rackliffe]

To be clear, I support the general agreement David and Jonathan seem to
have reached.

On Wed, May 3, 2023 at 3:05 PM Caleb Rackliffe 
wrote:

> Did we agree on a CQL syntax?
>
> On Wed, May 3, 2023 at 2:06 PM Rahul Xavier Singh <
> rahul.xavier.si...@gmail.com> wrote:
>
>> I like this approach. Thank you for those working on this vector search
>> initiative.
>>
>> Here's the feedback from my "user" hat for someone who is looking at
>> databases / indexes for my next LLM app.
>>
>> Can I take some python code and go from using an in memory vector store
>> like numpy or FAISS to something else? How easy is it for me to take my
>> python code and get it to work with this new external service which is no
>> longer just a library?
>> There's also tons of services that I can run on docker e.g. milvus,
>> redissearch, typesense, elasticsearch, opensearch and I may hit a hurdle
>> when trying to do a lot more data, so I look at Cassandra Vector Search.
>> Because I am familiar with SQL , Cassandra looks appealing since I can
>> potentially use "cql_agent" lib ( to be created for langchain and we're
>> looking into that now) or an existing CassandraVectorStore class?
>>
>> In most of these scenarios, if people are using langchain, llamaindex,
>> the underlying implementation is not as important since we shield the user
>> from CQL data types except at schema creation and most of this libs can be
>> opinionated and just suggest a generic schema.
>>
>> The ideal world is if I can just dump text into a field and do a natural
>> language query on it and have my DB do the embeddings for the document, and
>> then for the query for me. For now the libs can manage all that and they do
>> that well. We just need the interface to stay consistent and be relatively
>> easy to query in CQL. The most popular index in LLM retrieval augmented
>> patterns is pinecone. You make an index, you upsert, and then you query.
>> It's not assumed that you are also giving it content, though you can send
>> it metadata to have the document there.
>>
>> If we can have a similar workflow e.g. create a table with a vector type
>> OR create a table with an existing type and then add an index to it, no one
>> is going to sleep over it as long as it works. Having the ability to take a
>> table that has data, and then add a vector index doesn't make it any
>> different than adding a new field since I've got to calculate the
>> embeddings anyways.
>>
>> Would love to see how the CQL ends up looking like.
>> Rahul Singh
>>
>> Chief Executive Officer | Business Platform Architect m: 202.905.2818 e:
>> rahul.si...@anant.us li: http://linkedin.com/in/xingh ca:
>> http://calendly.com/xingh
>>
>> *We create, support, and manage real-time global data & analytics
>> platforms for the modern enterprise.*
>>
>> *Anant | https://anant.us <https://anant.us/>*
>>
>> 3 Washington Circle, Suite 301
>>
>> Washington, D.C. 20037
>>
>> *http://Cassandra.Link <http://cassandra.link/>* : The best resources
>> for Apache Cassandra
>>
>>
>> On Tue, May 2, 2023 at 6:39 PM Patrick McFadin 
>> wrote:
>>
>>> \o/
>>>
>>> Bring it in team. Group hug.
>>>
>>> Now if you'll excuse me, I'm going to go build my preso on how Cassandra
>>> is the only distributed database you can do vector search in an ACID
>>> transaction.
>>>
>>> Patrick
>>>
>>> On Tue, May 2, 2023 at 3:27 PM Jonathan Ellis  wrote:
>>>
>>>> I had a call with David.  We agreed that we want a "vector" data type
>>>> with these properties
>>>>
>>>> - Fixed length
>>>> - No nulls
>>>> - Random access not supported
>>>>
>>>> Where we disagreed was on my proposal to restrict vectors to only
>>>> numeric data.  David's points were that
>>>>
>>>> (1) He has a use case today for a data type with the other vector
>>>> properties,
>>>> (2) It doesn't seem reasonable to create two data types with the same
>>>> properties, one of which is restricted to numerics, and
>>>> (3) The restrictions that I want for numeric vectors make more sense at
>>>> the index and function level, than at the type level.
>>>>
>>>> I'm ready to concede that David has the better case here and move
>>>> forward with a vector implementat

Re: [POLL] Vector type for ML

[Caleb Rackliffe]

I actually still prefer *type[dimension]*, because I think I intuitively
read this as a primitive (meaning no null elements) array. Then we can have
the indexing apparatus only accept *frozen* for the HSNW case.

If that isn't intuitive to anyone else, I don't really have a strong
opinion...but...conflating "frozen" and "dense" seems like a bad idea. One
should indicate single vs. multi-cell, and the other the presence or
absence of nulls/zeros/whatever.

On Thu, May 4, 2023 at 12:51 PM Patrick McFadin  wrote:

> I agree with David's reasoning and the use of DENSE (and maybe eventually
> SPARSE). This is terminology well established in the data world, and it
> would lead to much easier adoption from users. VECTOR is close, but I can
> see having to create a lot of content around "How to use it and not get in
> trouble." (I have a lot of that content already)
>
>  - We don't have to explain what it is. A lot of prior art out there
> already [1][2][3]
>  - We're matching an established term with what users would expect. No
> surprises.
>  - Shorter ramp-up time for users. Cassandra is being modernized.
>
> The implementation is flexible, but the interface should empower our users
> to be awesome.
>
> Patrick
>
> 1 -
> https://stats.stackexchange.com/questions/266996/what-do-the-terms-dense-and-sparse-mean-in-the-context-of-neural-networks
> 2 -
> https://induraj2020.medium.com/what-are-sparse-features-and-dense-features-8d1746a77035
> 3 -
> https://revware.net/sparse-vs-dense-data-the-power-of-points-and-clouds/
>
> On Thu, May 4, 2023 at 10:25 AM David Capwell  wrote:
>
>> My views have changed over time on syntax and I feel type[dimention] may
>> not be the best, so it has gone lower in my own personal ranking… this is
>> my current preference
>>
>> 1) DENSE [dimention] | NON NULL [dimention]
>> 2) VECTOR
>> 3) type[dimention]
>>
>> My reasoning for this order
>>
>> * type[dimention] looks like syntax sugar for array, so
>> users may assume list/array semantics, but we limit to non-null elements in
>> a frozen array
>> * feel VECTOR as a prefix feels out of place, but VECTOR as a direct type
>> makes more sense… this also leads to a possible future of VECTOR
>> which is the non-fixed length version of this type.  What makes VECTOR
>> different from list/array?  non-null elements and is frozen.  I don’t feel
>> that VECTOR really tells users to expect non-null or frozen semantics, as
>> there exists different VECTOR types for those reasons (sparse vs dense)…
>> * DENSE may be confusing for people coming from languages where this just
>> means “sequential layout”, which is what our frozen array/list already are…
>> but since the target user is coming from a ML background, this shouldn’t
>> offer much confusion.  DENSE just means FROZEN in Cassandra, with NON NULL
>> elements (SPARSE allows for NULL and isn’t frozen)… So DENSE just acts as
>> syntax sugar for frozen
>>
>>
>> On May 4, 2023, at 4:13 AM, Brandon Williams  wrote:
>>
>> 1. VECTOR
>> 2. VECTOR FLOAT[n]
>> 3. FLOAT[N]   (Non null by default)
>>
>> Redundant or not, I think having the VECTOR keyword helps signify what
>> the app is generally about and helps get buy-in from ML stakeholders.
>>
>> On Thu, May 4, 2023 at 3:45 AM Benedict  wrote:
>>
>>
>> Hurrah for initial agreement.
>>
>> For syntax, I think one option was just FLOAT[N]. In VECTOR FLOAT[N],
>> VECTOR is redundant - FLOAT[N] is fully descriptive by itself. I don’t
>> think VECTOR should be used to simply imply non-null, as this would be very
>> unintuitive. More logical would be NONNULL, if this is the only condition
>> being applied. Alternatively for arrays we could default to NONNULL and
>> later introduce NULLABLE if we want to permit nulls.
>>
>> If the word vector is to be used it makes more sense to make it look like
>> a list, so VECTOR as here the word VECTOR is clearly not
>> redundant.
>>
>> So, I vote:
>>
>> 1) (NON NULL) FLOAT[N]
>> 2) FLOAT[N]   (Non null by default)
>> 3) VECTOR
>>
>>
>>
>> On 4 May 2023, at 08:52, Mick Semb Wever  wrote:
>>
>> 
>>
>>
>> Did we agree on a CQL syntax?
>>
>> I don’t believe there has been a pool on CQL syntax… my understanding
>> reading all the threads is that there are ~4-5 options and non are -1ed, so
>> believe we are waiting for majority rule on this?
>>
>>
>>
>>
>> Re-reading that thread, IIUC the valid choices remaining are…
>>
>> 1. VECTOR FLOAT[n]
>> 2. FLOAT VECTOR[n]
>> 3. VECTOR
>> 4. VECTOR[n]
>> 5. ARRAY
>> 6. NON-NULL FROZEN
>>
>>
>> Yes I'm putting my preference (1) first ;) because (banging on) if the
>> future of CQL will have FLOAT[n] and FROZEN, where the VECTOR
>> keyword is: for general cql users; just meaning "non-null and frozen",
>> these gel best together.
>>
>> Options (5) and (6) are for those that feel we can and should provide
>> this type without introducing the vector keyword.
>>
>>
>>
>>

Re: [POLL] Vector type for ML

[Caleb Rackliffe]

Even in the ML case, sparse can just mean zeros rather than nulls, and they
should compress similarly anyway.

If we really want null values, I'd rather leave that in collections space.

On Thu, May 4, 2023 at 8:59 PM Caleb Rackliffe 
wrote:

> I actually still prefer *type[dimension]*, because I think I intuitively
> read this as a primitive (meaning no null elements) array. Then we can have
> the indexing apparatus only accept *frozen* for the HSNW case.
>
> If that isn't intuitive to anyone else, I don't really have a strong
> opinion...but...conflating "frozen" and "dense" seems like a bad idea. One
> should indicate single vs. multi-cell, and the other the presence or
> absence of nulls/zeros/whatever.
>
> On Thu, May 4, 2023 at 12:51 PM Patrick McFadin 
> wrote:
>
>> I agree with David's reasoning and the use of DENSE (and maybe eventually
>> SPARSE). This is terminology well established in the data world, and it
>> would lead to much easier adoption from users. VECTOR is close, but I can
>> see having to create a lot of content around "How to use it and not get in
>> trouble." (I have a lot of that content already)
>>
>>  - We don't have to explain what it is. A lot of prior art out there
>> already [1][2][3]
>>  - We're matching an established term with what users would expect. No
>> surprises.
>>  - Shorter ramp-up time for users. Cassandra is being modernized.
>>
>> The implementation is flexible, but the interface should empower our
>> users to be awesome.
>>
>> Patrick
>>
>> 1 -
>> https://stats.stackexchange.com/questions/266996/what-do-the-terms-dense-and-sparse-mean-in-the-context-of-neural-networks
>> 2 -
>> https://induraj2020.medium.com/what-are-sparse-features-and-dense-features-8d1746a77035
>> 3 -
>> https://revware.net/sparse-vs-dense-data-the-power-of-points-and-clouds/
>>
>> On Thu, May 4, 2023 at 10:25 AM David Capwell  wrote:
>>
>>> My views have changed over time on syntax and I feel type[dimention] may
>>> not be the best, so it has gone lower in my own personal ranking… this is
>>> my current preference
>>>
>>> 1) DENSE [dimention] | NON NULL [dimention]
>>> 2) VECTOR
>>> 3) type[dimention]
>>>
>>> My reasoning for this order
>>>
>>> * type[dimention] looks like syntax sugar for array, so
>>> users may assume list/array semantics, but we limit to non-null elements in
>>> a frozen array
>>> * feel VECTOR as a prefix feels out of place, but VECTOR as a direct
>>> type makes more sense… this also leads to a possible future of VECTOR
>>> which is the non-fixed length version of this type.  What makes VECTOR
>>> different from list/array?  non-null elements and is frozen.  I don’t feel
>>> that VECTOR really tells users to expect non-null or frozen semantics, as
>>> there exists different VECTOR types for those reasons (sparse vs dense)…
>>> * DENSE may be confusing for people coming from languages where this
>>> just means “sequential layout”, which is what our frozen array/list already
>>> are… but since the target user is coming from a ML background, this
>>> shouldn’t offer much confusion.  DENSE just means FROZEN in Cassandra, with
>>> NON NULL elements (SPARSE allows for NULL and isn’t frozen)… So DENSE just
>>> acts as syntax sugar for frozen
>>>
>>>
>>> On May 4, 2023, at 4:13 AM, Brandon Williams  wrote:
>>>
>>> 1. VECTOR
>>> 2. VECTOR FLOAT[n]
>>> 3. FLOAT[N]   (Non null by default)
>>>
>>> Redundant or not, I think having the VECTOR keyword helps signify what
>>> the app is generally about and helps get buy-in from ML stakeholders.
>>>
>>> On Thu, May 4, 2023 at 3:45 AM Benedict  wrote:
>>>
>>>
>>> Hurrah for initial agreement.
>>>
>>> For syntax, I think one option was just FLOAT[N]. In VECTOR FLOAT[N],
>>> VECTOR is redundant - FLOAT[N] is fully descriptive by itself. I don’t
>>> think VECTOR should be used to simply imply non-null, as this would be very
>>> unintuitive. More logical would be NONNULL, if this is the only condition
>>> being applied. Alternatively for arrays we could default to NONNULL and
>>> later introduce NULLABLE if we want to permit nulls.
>>>
>>> If the word vector is to be used it makes more sense to make it look
>>> like a list, so VECTOR as here the word VECTOR is clearly not
>>> redundant.
>>>
>>> So, I vote:
>>>
&

Re: [POLL] Vector type for ML

[Caleb Rackliffe]

...where, just to be clear, VECTOR means a frozen fixed
size array w/ no null values?

On Fri, May 5, 2023 at 11:23 AM Jonathan Ellis  wrote:

> +10 for not inflicting unwieldy keywords on ML users.
>
> Re Josh's summary, mostly agreed, my only objection to adding the DENSE
> keyword is that I don't see a foreseeable future where we also support
> sparse vectors, so it would end up being unnecessary extra verbosity.  So
> my preference would be
>
> 1. VECTOR
> 2. DENSE VECTOR (space instead of underscore, SQL isn't
> afraid of spaces)
> 3. type[dimension]
>
> On Fri, May 5, 2023 at 10:54 AM Patrick McFadin 
> wrote:
>
>> I hope we are willing to consider developers that use our system because
>> if I had to teach people to use "NON-NULL FROZEN" I'm pretty sure
>> the response would be:
>>
>> Did you tell me to go write a distributed map-reduce job in Erlang? I
>> beleive I did, Bob.
>>
>> On Fri, May 5, 2023 at 8:05 AM Josh McKenzie 
>> wrote:
>>
>>> Idiomatically, to my mind, there's a question of "what space are we
>>> thinking about this datatype in"?
>>>
>>> - In the context of mathematics, nullability in a vector would be 0
>>> - In the context of Cassandra, nullability tends to mean a tombstone (or
>>> nothing)
>>> - In the context of programming languages, it's all over the place
>>>
>>> Given many models are exploring quantizing to int8 and other data types,
>>> there's definitely the "support other data types easily in the future"
>>> piece to me we need to keep in mind.
>>>
>>> So with the above and the "meet the user where they are and don't make
>>> them understand more of Cassandra than absolutely critical to use it", I
>>> lean:
>>>
>>> 1. DENSE_VECTOR
>>> 2. VECTOR
>>> 3. type[dimension]
>>>
>>> This leaves the path open for us to expand on it in the future with
>>> sparse support and allows us to introduce some semantics that indicate
>>> idioms around nullability for the users coming from a different space.
>>>
>>> "NON-NULL FROZEN" is strictly correct, however it requires
>>> understanding idioms of how Cassandra thinks about data (nulls mean
>>> different things to us, we have differences between frozen and non-frozen
>>> due to constraints in our storage engine and materialization of data, etc)
>>> that get in the way of users doing things in the pattern they're familiar
>>> with without learning more about the DB than they're probably looking to
>>> learn. Historically this has been a challenge for us in adoption; the
>>> classic "Why can't I just write and delete and write as much as I want? Why
>>> are deletes filling up my disk?" problem comes to mind.
>>>
>>> I'd also be happy with us supporting:
>>> * NON-NULL FROZEN
>>> * DENSE_VECTOR as syntactic sugar for the above
>>>
>>> If getting into the "built-in syntactic sugar mapping for communities
>>> and specific use-cases" is something we're willing to consider.
>>>
>>> On Fri, May 5, 2023, at 7:26 AM, Patrick McFadin wrote:
>>>
>>> I think we are still discussing implementation here when I'm talking
>>> about developer experience. I want developers to adopt this quickly, easily
>>> and be successful. Vector search is already a thing. People use it every
>>> day. A successful outcome, in my view, is developers picking up this
>>> feature without reading a manual. (Because they don't anyway and get in
>>> trouble) I did some more extensive research about what other DBs are using
>>> for syntax. The consensus is some variety of 'VECTOR', 'DENSE' and 'SPARSE'
>>>
>>> Pinecone[1] - dense_vector, sparse_vector
>>> Elastic[2]: dense_vector
>>> Milvus[3]: float_vector, binary_vector
>>> pgvector[4]: vector
>>> Weaviate[5]: Different approach. All typed arrays can be indexed
>>>
>>> Based on that I'm advocating a similar syntax:
>>>
>>> - DENSE VECTOR
>>> or
>>> - VECTOR
>>>
>>> [1] https://docs.pinecone.io/docs/hybrid-search
>>> [2]
>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
>>> [3] https://milvus.io/docs/create_collection.md
>>> [4] https://github.com/pgvector/pgvector
>>> [5] https://weaviate.io/develope

Re: [VOTE] CEP-29 CQL NOT Operator

[Caleb Rackliffe]

+1

On Tue, May 9, 2023 at 12:04 PM Piotr Kołaczkowski 
wrote:

> Let's vote.
>
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-29%3A+CQL+NOT+operator
>
> Piotr Kołaczkowski
> e. pkola...@datastax.com
> w. www.datastax.com
>

Re: CEP-30: Approximate Nearest Neighbor(ANN) Vector Search via Storage-Attached Indexes

[Caleb Rackliffe]

Anyone on this ML who still remembers DSE Search (or has experience w/
Elastic or SolrCloud) probably also knows that there are some significant
pieces of an optimized scatter/gather apparatus for IR (even without
sorting, which also doesn't exist yet) that do not exist in C* or it's
range query system (which SAI and all other 2i implementations use). SAI,
like all C* 2i implementations, is still a local index, and as that is the
case, anything built on it will perform best in partition-scoped (at least
on the read side) use-cases. (On the bright side, the project is moving
toward larger partitions being a possibility.) With smaller clusters or
use-cases that are extremely write-heavy/read-light, it's possible that the
full scatter/gather won't be too onerous, especially w/ a few small tweaks
(on top of a non-vnode cluster) to a.) keep fanout minimal and b.) keep
range/index queries to a single pass to minimize latency.

Whatever we do, we just need to avoid a situation down the road where users
don't understand these nuances and hit a wall where they try to use this in
a way that is fundamentally incompatible w/ the way the database
scales/works. (I've done my best to call this out in all discussions around
SAI over time, and there may even end up being further guardrails put in
place to make it even harder to misuse it...but I digress.)

Having said all that, I don't fundamentally have a problem w/ the proposal.

On Tue, May 9, 2023 at 2:11 PM Benedict  wrote:

> HNSW can in principle be made into a distributed index. But that would be
> quite a different paradigm to SAI.
>
> On 9 May 2023, at 19:30, Patrick McFadin  wrote:
>
> 
> Under the goals section, there is this line:
>
>
>1. Scatter/gather across replicas, combining topK from each to get
>global topK.
>
>
> But what I'm hearing is, exactly how will that happen? Maybe this is an
> SAI question too. How is that verified in SAI?
>
> On Tue, May 9, 2023 at 11:07 AM David Capwell  wrote:
>
>> Approach section doesn’t go over how this will handle cross replica
>> search, this would be good to flesh out… given results have a real ranking,
>> the current 2i logic may yield incorrect results… so would think we need
>> num_ranges / rf queries in the best case, with some new capability to sort
>> the results?  If my assumption is correct, then how errors are handled
>> should also be fleshed out… Example: 1k cluster without vnode and RF=3, so
>> 333 queries fanned out to match, then coordinator needs to sort… if 1 of
>> the queries fails and can’t fall back to peers… does the query fail (I
>> assume so)?
>>
>> On May 8, 2023, at 7:20 PM, Jonathan Ellis  wrote:
>>
>> Hi all,
>>
>> Following the recent discussion threads, I would like to propose CEP-30
>> to add Approximate Nearest Neighbor (ANN) Vector Search via
>> Storage-Attached Indexes (SAI) to Apache Cassandra.
>>
>> The primary goal of this proposal is to implement ANN vector search
>> capabilities, making Cassandra more useful to AI developers and
>> organizations managing large datasets that can benefit from fast similarity
>> search.
>>
>> The implementation will leverage Lucene's Hierarchical Navigable Small
>> World (HNSW) library and introduce a new CQL data type for vector
>> embeddings, a new SAI index for ANN search functionality, and a new CQL
>> operator for performing ANN search queries.
>>
>> We are targeting the 5.0 release for this feature, in conjunction with
>> the release of SAI. The proposed changes will maintain compatibility with
>> existing Cassandra functionality and compose well with the already-approved
>> SAI features.
>>
>> Please find the full CEP document here:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes
>>
>> --
>> Jonathan Ellis
>> co-founder, http://www.datastax.com
>> @spyced
>>
>>
>>

[DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

Earlier today, Mick started a thread on the future of our index creation
DDL on Slack:

https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019

At the moment, there are two ways to create a secondary index.

*1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()*

This creates an optionally named legacy 2i on the provided table and column.

ex. CREATE INDEX my_index ON kd.tbl(my_text_col)

*2.) CREATE CUSTOM INDEX [IF NOT EXISTS] [name] ON  () USING
 [WITH OPTIONS = ]*

This creates a secondary index on the provided table and column using the
specified 2i implementation class and (optional) parameters.

ex. CREATE CUSTOM INDEX my_index ON ks.tbl(my_text_col) USING
'StorageAttachedIndex'

(Note that the work on SAI added aliasing, so `StorageAttachedIndex` is
shorthand for the fully-qualified class name, which is also valid.)

So what is there to discuss?

The concern Mick raised is...

"...just folk continuing to use CREATE INDEX  because they think CREATE
CUSTOM INDEX is advanced (or just don't know of it), and we leave users
doing 2i (when they think they are, and/or we definitely want them to be,
using SAI)"

To paraphrase, we want people to use SAI once it's available where
possible, and the default behavior of CREATE INDEX could be at odds w/ that.

The proposal we seem to have landed on is something like the following:

For 5.0:

1.) Disable by default the creation of new legacy 2i via CREATE INDEX.
2.) Leave CREATE CUSTOM INDEX...USING... available by default.

(Note: How this would interact w/ the existing secondary_indexes_enabled
YAML options isn't clear yet.)

Post-5.0:

1.) Deprecate and eventually remove SASI when SAI hits full feature parity
w/ it.
2.) Replace both CREATE INDEX and CREATE CUSTOM INDEX w/ something of a
hybrid between the two. For example, CREATE INDEX...USING...WITH. This
would both be flexible enough to accommodate index implementation selection
and prescriptive enough to force the user to make a decision (and wouldn't
change the legacy behavior of the existing CREATE INDEX). In this world,
creating a legacy 2i might look something like CREATE INDEX...USING `legacy`
.
3.) Eventually deprecate CREATE CUSTOM INDEX...USING.

Eventually we would have a single enabled DDL statement for index creation
that would be minimal but also explicit/able to handle some evolution.

What does everyone think?

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

I see a broad desire here to have a configurable (YAML) default
implementation for CREATE INDEX. I'm not strongly opposed to that, as the
concept of a default index implementation is pretty standard for most DBMS
(see Postgres, etc.). However, keep in mind that if we do that, we still
need to either revert to CREATE CUSTOM INDEX or add the USING...WITH...
extensions to CREATE INDEX to override the default or specify parameters,
which will be in play once SAI supports basic text tokenization/filtering.
Having to revert to CREATE CUSTOM INDEX sounds pretty awful, so I'd prefer
allowing USING...WITH... for CREATE INDEX and just deprecating CREATE
CUSTOM INDEX (at least after 5.0), but that's more or less what my original
proposal was above (modulo the configurable default).

Thoughts?

On Wed, May 10, 2023 at 2:59 AM Benedict  wrote:

> I’m not convinced by the changing defaults argument here. The
> characteristics of the two index types are very different, and users with
> scripts that make indexes today shouldn’t have their behaviour change.
>
> We could introduce new syntax that properly appreciates there’s no default
> index, perhaps CREATE LOCAL [type] INDEX? To also make clear that these
> indexes involve a partition key or scatter gather
>
> On 10 May 2023, at 06:26, guo Maxwell  wrote:
>
> 
> +1 , as we must Improve the image of your own default indexing ability.
>
> and As for *CREATE CUSTOM INDEX *, should we just left as it is and we
> can disable the ability for create SAI through  *CREATE CUSTOM INDEX*  in
> some version after 5.0?
>
> for as I know there may be users using this as a plugin-index interface,
> like https://github.com/Stratio/cassandra-lucene-index (though these
> project may be inactive， But if someone wants to do something similar in
> the future, we don't have to stop).
>
>
>
> Jonathan Ellis  于2023年5月10日周三 10:01写道：
>
>> +1 for this, especially in the long term.  CREATE INDEX should do the
>> right thing for most people without requiring extra ceremony.
>>
>> On Tue, May 9, 2023 at 5:20 PM Jeremiah D Jordan <
>> jeremiah.jor...@gmail.com> wrote:
>>
>>> If the consensus is that SAI is the right default index, then we should
>>> just change CREATE INDEX to be SAI, and legacy 2i to be a CUSTOM INDEX.
>>>
>>>
>>> On May 9, 2023, at 4:44 PM, Caleb Rackliffe 
>>> wrote:
>>>
>>> Earlier today, Mick started a thread on the future of our index creation
>>> DDL on Slack:
>>>
>>> https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019
>>>
>>> At the moment, there are two ways to create a secondary index.
>>>
>>> *1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()*
>>>
>>> This creates an optionally named legacy 2i on the provided table and
>>> column.
>>>
>>> ex. CREATE INDEX my_index ON kd.tbl(my_text_col)
>>>
>>> *2.) CREATE CUSTOM INDEX [IF NOT EXISTS] [name] ON  ()
>>> USING  [WITH OPTIONS = ]*
>>>
>>> This creates a secondary index on the provided table and column using
>>> the specified 2i implementation class and (optional) parameters.
>>>
>>> ex. CREATE CUSTOM INDEX my_index ON ks.tbl(my_text_col) USING
>>> 'StorageAttachedIndex'
>>>
>>> (Note that the work on SAI added aliasing, so `StorageAttachedIndex` is
>>> shorthand for the fully-qualified class name, which is also valid.)
>>>
>>> So what is there to discuss?
>>>
>>> The concern Mick raised is...
>>>
>>> "...just folk continuing to use CREATE INDEX  because they think CREATE
>>> CUSTOM INDEX is advanced (or just don't know of it), and we leave users
>>> doing 2i (when they think they are, and/or we definitely want them to be,
>>> using SAI)"
>>>
>>> To paraphrase, we want people to use SAI once it's available where
>>> possible, and the default behavior of CREATE INDEX could be at odds w/
>>> that.
>>>
>>> The proposal we seem to have landed on is something like the following:
>>>
>>> For 5.0:
>>>
>>> 1.) Disable by default the creation of new legacy 2i via CREATE INDEX.
>>> 2.) Leave CREATE CUSTOM INDEX...USING... available by default.
>>>
>>> (Note: How this would interact w/ the existing secondary_indexes_enabled
>>> YAML options isn't clear yet.)
>>>
>>> Post-5.0:
>>>
>>> 1.) Deprecate and eventually remove SASI when SAI hits full feature
>>> parity w/ it.
>>> 2.) Replace both CREATE INDEX and CRE

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

> We could introduce new syntax that properly appreciates there’s no
default index, perhaps CREATE LOCAL [type] INDEX? To also make clear that
these indexes involve a partition key or scatter gather

I think this is something we should handle in guardrails space on the query
side for all indexes. Specifically, we should have the ability to diallow
scatter/gather queries against indexes (and *all* indexes are local rn).
Mentioning this at the DDL level probably isn't necessary.

On Wed, May 10, 2023 at 2:59 AM Benedict  wrote:

> I’m not convinced by the changing defaults argument here. The
> characteristics of the two index types are very different, and users with
> scripts that make indexes today shouldn’t have their behaviour change.
>
> We could introduce new syntax that properly appreciates there’s no default
> index, perhaps CREATE LOCAL [type] INDEX? To also make clear that these
> indexes involve a partition key or scatter gather
>
> On 10 May 2023, at 06:26, guo Maxwell  wrote:
>
> 
> +1 , as we must Improve the image of your own default indexing ability.
>
> and As for *CREATE CUSTOM INDEX *, should we just left as it is and we
> can disable the ability for create SAI through  *CREATE CUSTOM INDEX*  in
> some version after 5.0?
>
> for as I know there may be users using this as a plugin-index interface,
> like https://github.com/Stratio/cassandra-lucene-index (though these
> project may be inactive， But if someone wants to do something similar in
> the future, we don't have to stop).
>
>
>
> Jonathan Ellis  于2023年5月10日周三 10:01写道：
>
>> +1 for this, especially in the long term.  CREATE INDEX should do the
>> right thing for most people without requiring extra ceremony.
>>
>> On Tue, May 9, 2023 at 5:20 PM Jeremiah D Jordan <
>> jeremiah.jor...@gmail.com> wrote:
>>
>>> If the consensus is that SAI is the right default index, then we should
>>> just change CREATE INDEX to be SAI, and legacy 2i to be a CUSTOM INDEX.
>>>
>>>
>>> On May 9, 2023, at 4:44 PM, Caleb Rackliffe 
>>> wrote:
>>>
>>> Earlier today, Mick started a thread on the future of our index creation
>>> DDL on Slack:
>>>
>>> https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019
>>>
>>> At the moment, there are two ways to create a secondary index.
>>>
>>> *1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()*
>>>
>>> This creates an optionally named legacy 2i on the provided table and
>>> column.
>>>
>>> ex. CREATE INDEX my_index ON kd.tbl(my_text_col)
>>>
>>> *2.) CREATE CUSTOM INDEX [IF NOT EXISTS] [name] ON  ()
>>> USING  [WITH OPTIONS = ]*
>>>
>>> This creates a secondary index on the provided table and column using
>>> the specified 2i implementation class and (optional) parameters.
>>>
>>> ex. CREATE CUSTOM INDEX my_index ON ks.tbl(my_text_col) USING
>>> 'StorageAttachedIndex'
>>>
>>> (Note that the work on SAI added aliasing, so `StorageAttachedIndex` is
>>> shorthand for the fully-qualified class name, which is also valid.)
>>>
>>> So what is there to discuss?
>>>
>>> The concern Mick raised is...
>>>
>>> "...just folk continuing to use CREATE INDEX  because they think CREATE
>>> CUSTOM INDEX is advanced (or just don't know of it), and we leave users
>>> doing 2i (when they think they are, and/or we definitely want them to be,
>>> using SAI)"
>>>
>>> To paraphrase, we want people to use SAI once it's available where
>>> possible, and the default behavior of CREATE INDEX could be at odds w/
>>> that.
>>>
>>> The proposal we seem to have landed on is something like the following:
>>>
>>> For 5.0:
>>>
>>> 1.) Disable by default the creation of new legacy 2i via CREATE INDEX.
>>> 2.) Leave CREATE CUSTOM INDEX...USING... available by default.
>>>
>>> (Note: How this would interact w/ the existing secondary_indexes_enabled
>>> YAML options isn't clear yet.)
>>>
>>> Post-5.0:
>>>
>>> 1.) Deprecate and eventually remove SASI when SAI hits full feature
>>> parity w/ it.
>>> 2.) Replace both CREATE INDEX and CREATE CUSTOM INDEX w/ something of a
>>> hybrid between the two. For example, CREATE INDEX...USING...WITH. This
>>> would both be flexible enough to accommodate index implementation selection
>>> and prescriptive enough to force the user to make a decision (and wouldn't
>>> change the legacy behavior of the existing CREATE INDEX). In this
>>> world, creating a legacy 2i might look something like CREATE
>>> INDEX...USING `legacy`.
>>> 3.) Eventually deprecate CREATE CUSTOM INDEX...USING.
>>>
>>> Eventually we would have a single enabled DDL statement for index
>>> creation that would be minimal but also explicit/able to handle some
>>> evolution.
>>>
>>> What does everyone think?
>>>
>>>
>>>
>>
>> --
>> Jonathan Ellis
>> co-founder, http://www.datastax.com
>> @spyced
>>
>
>
> --
> you are the apple of my eye !
>
>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

tl;dr If you take my original proposal and change only the fact that CREATE
INDEX retains a configurable default, I think we get to the same place?

(Then it's just a matter of what we do in 5.0 vs. after 5.0...)

On Wed, May 10, 2023 at 11:00 AM Caleb Rackliffe 
wrote:

> I see a broad desire here to have a configurable (YAML) default
> implementation for CREATE INDEX. I'm not strongly opposed to that, as the
> concept of a default index implementation is pretty standard for most DBMS
> (see Postgres, etc.). However, keep in mind that if we do that, we still
> need to either revert to CREATE CUSTOM INDEX or add the USING...WITH...
> extensions to CREATE INDEX to override the default or specify parameters,
> which will be in play once SAI supports basic text tokenization/filtering.
> Having to revert to CREATE CUSTOM INDEX sounds pretty awful, so I'd
> prefer allowing USING...WITH... for CREATE INDEX and just deprecating CREATE
> CUSTOM INDEX (at least after 5.0), but that's more or less what my
> original proposal was above (modulo the configurable default).
>
> Thoughts?
>
> On Wed, May 10, 2023 at 2:59 AM Benedict  wrote:
>
>> I’m not convinced by the changing defaults argument here. The
>> characteristics of the two index types are very different, and users with
>> scripts that make indexes today shouldn’t have their behaviour change.
>>
>> We could introduce new syntax that properly appreciates there’s no
>> default index, perhaps CREATE LOCAL [type] INDEX? To also make clear that
>> these indexes involve a partition key or scatter gather
>>
>> On 10 May 2023, at 06:26, guo Maxwell  wrote:
>>
>> 
>> +1 , as we must Improve the image of your own default indexing ability.
>>
>> and As for *CREATE CUSTOM INDEX *, should we just left as it is and we
>> can disable the ability for create SAI through  *CREATE CUSTOM INDEX*  in
>> some version after 5.0?
>>
>> for as I know there may be users using this as a plugin-index interface,
>> like https://github.com/Stratio/cassandra-lucene-index (though these
>> project may be inactive， But if someone wants to do something similar in
>> the future, we don't have to stop).
>>
>>
>>
>> Jonathan Ellis  于2023年5月10日周三 10:01写道：
>>
>>> +1 for this, especially in the long term.  CREATE INDEX should do the
>>> right thing for most people without requiring extra ceremony.
>>>
>>> On Tue, May 9, 2023 at 5:20 PM Jeremiah D Jordan <
>>> jeremiah.jor...@gmail.com> wrote:
>>>
>>>> If the consensus is that SAI is the right default index, then we should
>>>> just change CREATE INDEX to be SAI, and legacy 2i to be a CUSTOM INDEX.
>>>>
>>>>
>>>> On May 9, 2023, at 4:44 PM, Caleb Rackliffe 
>>>> wrote:
>>>>
>>>> Earlier today, Mick started a thread on the future of our index
>>>> creation DDL on Slack:
>>>>
>>>> https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019
>>>>
>>>> At the moment, there are two ways to create a secondary index.
>>>>
>>>> *1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()*
>>>>
>>>> This creates an optionally named legacy 2i on the provided table and
>>>> column.
>>>>
>>>> ex. CREATE INDEX my_index ON kd.tbl(my_text_col)
>>>>
>>>> *2.) CREATE CUSTOM INDEX [IF NOT EXISTS] [name] ON  ()
>>>> USING  [WITH OPTIONS = ]*
>>>>
>>>> This creates a secondary index on the provided table and column using
>>>> the specified 2i implementation class and (optional) parameters.
>>>>
>>>> ex. CREATE CUSTOM INDEX my_index ON ks.tbl(my_text_col) USING
>>>> 'StorageAttachedIndex'
>>>>
>>>> (Note that the work on SAI added aliasing, so `StorageAttachedIndex`
>>>> is shorthand for the fully-qualified class name, which is also valid.)
>>>>
>>>> So what is there to discuss?
>>>>
>>>> The concern Mick raised is...
>>>>
>>>> "...just folk continuing to use CREATE INDEX  because they think CREATE
>>>> CUSTOM INDEX is advanced (or just don't know of it), and we leave
>>>> users doing 2i (when they think they are, and/or we definitely want them to
>>>> be, using SAI)"
>>>>
>>>> To paraphrase, we want people to use SAI once it's available where
>>>> possible, and the default behavior of CREATE INDEX could be at odds w/
>>>> t

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

> This creates huge headaches for everyone successfully using 2i today
though, and SAI *is not* guaranteed to perform as well or better - it has a
very different performance profile.

We wouldn't have even advanced it to this point if we didn't have copious
amounts of (not all public, I know, working on it) evidence it did for the
vast majority of workloads. Having said that, I don't strongly agree that
we should make it the default in 5.0, because performance isn't the only
concern. (correctness, DDL back-compat, which we've sort of touched w/ the
YAML default option, etc.)

This conversation is now going in like 3 different directions, or at least
3 different "packages" of ideas, so there isn't even a single thing to vote
on. Let me read through again and try to distill into something that we
might be able to do so with...

On Fri, May 12, 2023 at 7:56 AM Aleksey Yeshchenko 
wrote:

> This.
>
> I would also consider adding CREATE LEGACY INDEX syntax as an alias for
> today’s CREATE INDEX, the latter to be deprecated and (in very distant
> future) removed.
>
> On 12 May 2023, at 13:14, Benedict  wrote:
>
> This creates huge headaches for everyone successfully using 2i today
> though, and SAI *is not* guaranteed to perform as well or better - it has a
> very different performance profile.
>
> I think we should deprecate CREATE INDEX, and introduce new syntax CREATE
> LOCAL INDEX to make clear that this is not a global index, and that this
> should require the USING syntax to avoid this problem in future.
>
> We should report warnings to the client when CREATE INDEX is used,
> indicating it is deprecated.
>
>
>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

We don't need to know everything about SAI's performance profile to plan
and execute some small, reasonable things now for 5.0. I'm going to try to
summarize the least controversial package of ideas from the discussion
above. I've left out creating any new syntax. For example, I think CREATE
LOCAL INDEX, while explicit, is just not necessary. We don't use CREATE
LOCAL TABLE, although it has the same locality as our indexes.

Okay, so the proposal for 5.0...

1.) Add a YAML option that specifies a default implementation for CREATE
INDEX, and make this the legacy 2i for now. No existing DDL breaks. We
don't have to commit to the absolute superiority of SAI.
2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go to
market using CREATE CUSTOM INDEX, which feels...not so polished. (The
backend for this already exists w/ CREATE CUSTOM INDEX.)
3.) Leave in place but deprecate (client warnings could work?) CREATE
CUSTOM INDEX. Support the syntax for the foreseeable future.

Can we live w/ this?

I don't think any information about SAI we could possibly acquire before a
5.0 release would affect the reasonableness of this much.


On Fri, May 12, 2023 at 10:54 AM Benedict  wrote:

> if we didn't have copious amounts of (not all public, I know, working on
> it) evidence
>
>
> If that’s the assumption on which this proposal is based, let’s discuss
> the evidence base first, as given the fundamentally different way they work
> (almost diametrically opposite), I would want to see a very high quality of
> evidence to support the claim.
>
> I don’t think we can resolve this conversation effectively until this
> question is settled.
>
> On 12 May 2023, at 16:19, Caleb Rackliffe 
> wrote:
>
> 
> > This creates huge headaches for everyone successfully using 2i today
> though, and SAI *is not* guaranteed to perform as well or better - it has a
> very different performance profile.
>
> We wouldn't have even advanced it to this point if we didn't have copious
> amounts of (not all public, I know, working on it) evidence it did for the
> vast majority of workloads. Having said that, I don't strongly agree that
> we should make it the default in 5.0, because performance isn't the only
> concern. (correctness, DDL back-compat, which we've sort of touched w/ the
> YAML default option, etc.)
>
> This conversation is now going in like 3 different directions, or at least
> 3 different "packages" of ideas, so there isn't even a single thing to vote
> on. Let me read through again and try to distill into something that we
> might be able to do so with...
>
> On Fri, May 12, 2023 at 7:56 AM Aleksey Yeshchenko 
> wrote:
>
>> This.
>>
>> I would also consider adding CREATE LEGACY INDEX syntax as an alias for
>> today’s CREATE INDEX, the latter to be deprecated and (in very distant
>> future) removed.
>>
>> On 12 May 2023, at 13:14, Benedict  wrote:
>>
>> This creates huge headaches for everyone successfully using 2i today
>> though, and SAI *is not* guaranteed to perform as well or better - it has a
>> very different performance profile.
>>
>> I think we should deprecate CREATE INDEX, and introduce new syntax CREATE
>> LOCAL INDEX to make clear that this is not a global index, and that this
>> should require the USING syntax to avoid this problem in future.
>>
>> We should report warnings to the client when CREATE INDEX is used,
>> indicating it is deprecated.
>>
>>
>>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

...and if we decide before the 5.0 release that we have enough information
to change the default (#1), we can change it in a matter of minutes.

On Fri, May 12, 2023 at 11:28 AM Caleb Rackliffe 
wrote:

> We don't need to know everything about SAI's performance profile to plan
> and execute some small, reasonable things now for 5.0. I'm going to try to
> summarize the least controversial package of ideas from the discussion
> above. I've left out creating any new syntax. For example, I think CREATE
> LOCAL INDEX, while explicit, is just not necessary. We don't use CREATE
> LOCAL TABLE, although it has the same locality as our indexes.
>
> Okay, so the proposal for 5.0...
>
> 1.) Add a YAML option that specifies a default implementation for CREATE
> INDEX, and make this the legacy 2i for now. No existing DDL breaks. We
> don't have to commit to the absolute superiority of SAI.
> 2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go
> to market using CREATE CUSTOM INDEX, which feels...not so polished. (The
> backend for this already exists w/ CREATE CUSTOM INDEX.)
> 3.) Leave in place but deprecate (client warnings could work?) CREATE
> CUSTOM INDEX. Support the syntax for the foreseeable future.
>
> Can we live w/ this?
>
> I don't think any information about SAI we could possibly acquire before a
> 5.0 release would affect the reasonableness of this much.
>
>
> On Fri, May 12, 2023 at 10:54 AM Benedict  wrote:
>
>> if we didn't have copious amounts of (not all public, I know, working on
>> it) evidence
>>
>>
>> If that’s the assumption on which this proposal is based, let’s discuss
>> the evidence base first, as given the fundamentally different way they work
>> (almost diametrically opposite), I would want to see a very high quality of
>> evidence to support the claim.
>>
>> I don’t think we can resolve this conversation effectively until this
>> question is settled.
>>
>> On 12 May 2023, at 16:19, Caleb Rackliffe 
>> wrote:
>>
>> 
>> > This creates huge headaches for everyone successfully using 2i today
>> though, and SAI *is not* guaranteed to perform as well or better - it has a
>> very different performance profile.
>>
>> We wouldn't have even advanced it to this point if we didn't have copious
>> amounts of (not all public, I know, working on it) evidence it did for the
>> vast majority of workloads. Having said that, I don't strongly agree that
>> we should make it the default in 5.0, because performance isn't the only
>> concern. (correctness, DDL back-compat, which we've sort of touched w/ the
>> YAML default option, etc.)
>>
>> This conversation is now going in like 3 different directions, or at
>> least 3 different "packages" of ideas, so there isn't even a single thing
>> to vote on. Let me read through again and try to distill into something
>> that we might be able to do so with...
>>
>> On Fri, May 12, 2023 at 7:56 AM Aleksey Yeshchenko 
>> wrote:
>>
>>> This.
>>>
>>> I would also consider adding CREATE LEGACY INDEX syntax as an alias for
>>> today’s CREATE INDEX, the latter to be deprecated and (in very distant
>>> future) removed.
>>>
>>> On 12 May 2023, at 13:14, Benedict  wrote:
>>>
>>> This creates huge headaches for everyone successfully using 2i today
>>> though, and SAI *is not* guaranteed to perform as well or better - it has a
>>> very different performance profile.
>>>
>>> I think we should deprecate CREATE INDEX, and introduce new syntax
>>> CREATE LOCAL INDEX to make clear that this is not a global index, and that
>>> this should require the USING syntax to avoid this problem in future.
>>>
>>> We should report warnings to the client when CREATE INDEX is used,
>>> indicating it is deprecated.
>>>
>>>
>>>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

I don't particularly like the YAML solution either, but absent that, we're
back to fighting about whether we introduce entirely new syntax or hard cut
over to SAI at some point.

We already have per-node configuration in the YAML that determines whether
or not we can create a 2i at all, right?

What if we just do #2 and #3 and punt on everything else?

On Fri, May 12, 2023 at 11:56 AM Benedict  wrote:

> A table is not a local concept at all, it has a global primary index -
> that’s the core idea of Cassandra.
>
> I agree with Brandon that changing CQL behaviour like this based on node
> config is really not ideal. New syntax is by far the simplest and safest
> solution to this IMO. It doesn’t have to use the word LOCAL, but I think
> that’s anyway an improvement, personally.
>
> In future we will hopefully offer GLOBAL indexes, and IMO it is better to
> reify the distinction in the syntax.
>
> On 12 May 2023, at 17:29, Caleb Rackliffe 
> wrote:
>
> 
> We don't need to know everything about SAI's performance profile to plan
> and execute some small, reasonable things now for 5.0. I'm going to try to
> summarize the least controversial package of ideas from the discussion
> above. I've left out creating any new syntax. For example, I think CREATE
> LOCAL INDEX, while explicit, is just not necessary. We don't use CREATE
> LOCAL TABLE, although it has the same locality as our indexes.
>
> Okay, so the proposal for 5.0...
>
> 1.) Add a YAML option that specifies a default implementation for CREATE
> INDEX, and make this the legacy 2i for now. No existing DDL breaks. We
> don't have to commit to the absolute superiority of SAI.
> 2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go
> to market using CREATE CUSTOM INDEX, which feels...not so polished. (The
> backend for this already exists w/ CREATE CUSTOM INDEX.)
> 3.) Leave in place but deprecate (client warnings could work?) CREATE
> CUSTOM INDEX. Support the syntax for the foreseeable future.
>
> Can we live w/ this?
>
> I don't think any information about SAI we could possibly acquire before a
> 5.0 release would affect the reasonableness of this much.
>
>
> On Fri, May 12, 2023 at 10:54 AM Benedict  wrote:
>
>> if we didn't have copious amounts of (not all public, I know, working on
>> it) evidence
>>
>>
>> If that’s the assumption on which this proposal is based, let’s discuss
>> the evidence base first, as given the fundamentally different way they work
>> (almost diametrically opposite), I would want to see a very high quality of
>> evidence to support the claim.
>>
>> I don’t think we can resolve this conversation effectively until this
>> question is settled.
>>
>> On 12 May 2023, at 16:19, Caleb Rackliffe 
>> wrote:
>>
>> 
>> > This creates huge headaches for everyone successfully using 2i today
>> though, and SAI *is not* guaranteed to perform as well or better - it has a
>> very different performance profile.
>>
>> We wouldn't have even advanced it to this point if we didn't have copious
>> amounts of (not all public, I know, working on it) evidence it did for the
>> vast majority of workloads. Having said that, I don't strongly agree that
>> we should make it the default in 5.0, because performance isn't the only
>> concern. (correctness, DDL back-compat, which we've sort of touched w/ the
>> YAML default option, etc.)
>>
>> This conversation is now going in like 3 different directions, or at
>> least 3 different "packages" of ideas, so there isn't even a single thing
>> to vote on. Let me read through again and try to distill into something
>> that we might be able to do so with...
>>
>> On Fri, May 12, 2023 at 7:56 AM Aleksey Yeshchenko 
>> wrote:
>>
>>> This.
>>>
>>> I would also consider adding CREATE LEGACY INDEX syntax as an alias for
>>> today’s CREATE INDEX, the latter to be deprecated and (in very distant
>>> future) removed.
>>>
>>> On 12 May 2023, at 13:14, Benedict  wrote:
>>>
>>> This creates huge headaches for everyone successfully using 2i today
>>> though, and SAI *is not* guaranteed to perform as well or better - it has a
>>> very different performance profile.
>>>
>>> I think we should deprecate CREATE INDEX, and introduce new syntax
>>> CREATE LOCAL INDEX to make clear that this is not a global index, and that
>>> this should require the USING syntax to avoid this problem in future.
>>>
>>> We should report warnings to the client when CREATE INDEX is used,
>>> indicating it is deprecated.
>>>
>>>
>>>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

I don't want to cut over for 5.0 either way. I was more contrasting a
configurable cutover in 5.0 vs. a hard cutover later.

On Fri, May 12, 2023 at 12:09 PM Benedict  wrote:

> If the performance characteristics are as clear cut as you think, then
> maybe it will be an easy decision once the evidence is available for
> everyone to consider?
>
> If not, then we probably can’t do the hard cutover and so the answer is
> still pretty simple?
>
> On 12 May 2023, at 18:04, Caleb Rackliffe 
> wrote:
>
> 
> I don't particularly like the YAML solution either, but absent that, we're
> back to fighting about whether we introduce entirely new syntax or hard cut
> over to SAI at some point.
>
> We already have per-node configuration in the YAML that determines whether
> or not we can create a 2i at all, right?
>
> What if we just do #2 and #3 and punt on everything else?
>
> On Fri, May 12, 2023 at 11:56 AM Benedict  wrote:
>
>> A table is not a local concept at all, it has a global primary index -
>> that’s the core idea of Cassandra.
>>
>> I agree with Brandon that changing CQL behaviour like this based on node
>> config is really not ideal. New syntax is by far the simplest and safest
>> solution to this IMO. It doesn’t have to use the word LOCAL, but I think
>> that’s anyway an improvement, personally.
>>
>> In future we will hopefully offer GLOBAL indexes, and IMO it is better to
>> reify the distinction in the syntax.
>>
>> On 12 May 2023, at 17:29, Caleb Rackliffe 
>> wrote:
>>
>> 
>> We don't need to know everything about SAI's performance profile to plan
>> and execute some small, reasonable things now for 5.0. I'm going to try to
>> summarize the least controversial package of ideas from the discussion
>> above. I've left out creating any new syntax. For example, I think CREATE
>> LOCAL INDEX, while explicit, is just not necessary. We don't use CREATE
>> LOCAL TABLE, although it has the same locality as our indexes.
>>
>> Okay, so the proposal for 5.0...
>>
>> 1.) Add a YAML option that specifies a default implementation for CREATE
>> INDEX, and make this the legacy 2i for now. No existing DDL breaks. We
>> don't have to commit to the absolute superiority of SAI.
>> 2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go
>> to market using CREATE CUSTOM INDEX, which feels...not so polished. (The
>> backend for this already exists w/ CREATE CUSTOM INDEX.)
>> 3.) Leave in place but deprecate (client warnings could work?) CREATE
>> CUSTOM INDEX. Support the syntax for the foreseeable future.
>>
>> Can we live w/ this?
>>
>> I don't think any information about SAI we could possibly acquire before
>> a 5.0 release would affect the reasonableness of this much.
>>
>>
>> On Fri, May 12, 2023 at 10:54 AM Benedict  wrote:
>>
>>> if we didn't have copious amounts of (not all public, I know, working on
>>> it) evidence
>>>
>>>
>>> If that’s the assumption on which this proposal is based, let’s discuss
>>> the evidence base first, as given the fundamentally different way they work
>>> (almost diametrically opposite), I would want to see a very high quality of
>>> evidence to support the claim.
>>>
>>> I don’t think we can resolve this conversation effectively until this
>>> question is settled.
>>>
>>> On 12 May 2023, at 16:19, Caleb Rackliffe 
>>> wrote:
>>>
>>> 
>>> > This creates huge headaches for everyone successfully using 2i today
>>> though, and SAI *is not* guaranteed to perform as well or better - it has a
>>> very different performance profile.
>>>
>>> We wouldn't have even advanced it to this point if we didn't have
>>> copious amounts of (not all public, I know, working on it) evidence it did
>>> for the vast majority of workloads. Having said that, I don't strongly
>>> agree that we should make it the default in 5.0, because performance isn't
>>> the only concern. (correctness, DDL back-compat, which we've sort of
>>> touched w/ the YAML default option, etc.)
>>>
>>> This conversation is now going in like 3 different directions, or at
>>> least 3 different "packages" of ideas, so there isn't even a single thing
>>> to vote on. Let me read through again and try to distill into something
>>> that we might be able to do so with...
>>>
>>> On Fri, May 12, 2023 at 7:56 AM Aleksey Yeshchenko 

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

So the weakest version of the plan that actually accomplishes something
useful for 5.0:

1.) Just leave the CREATE INDEX default alone for now. Hard switch the
default after 5.0.
2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go to
market using CREATE CUSTOM INDEX, which feels...not so polished. (The
backend for this already exists w/ CREATE CUSTOM INDEX.)
3.) Leave in place but deprecate (client warnings could work?) CREATE
CUSTOM INDEX. Support the syntax for the foreseeable future.

Any objections to that?

On Fri, May 12, 2023 at 12:10 PM Caleb Rackliffe 
wrote:

> I don't want to cut over for 5.0 either way. I was more contrasting a
> configurable cutover in 5.0 vs. a hard cutover later.
>
> On Fri, May 12, 2023 at 12:09 PM Benedict  wrote:
>
>> If the performance characteristics are as clear cut as you think, then
>> maybe it will be an easy decision once the evidence is available for
>> everyone to consider?
>>
>> If not, then we probably can’t do the hard cutover and so the answer is
>> still pretty simple?
>>
>> On 12 May 2023, at 18:04, Caleb Rackliffe 
>> wrote:
>>
>> 
>> I don't particularly like the YAML solution either, but absent that,
>> we're back to fighting about whether we introduce entirely new syntax or
>> hard cut over to SAI at some point.
>>
>> We already have per-node configuration in the YAML that determines
>> whether or not we can create a 2i at all, right?
>>
>> What if we just do #2 and #3 and punt on everything else?
>>
>> On Fri, May 12, 2023 at 11:56 AM Benedict  wrote:
>>
>>> A table is not a local concept at all, it has a global primary index -
>>> that’s the core idea of Cassandra.
>>>
>>> I agree with Brandon that changing CQL behaviour like this based on node
>>> config is really not ideal. New syntax is by far the simplest and safest
>>> solution to this IMO. It doesn’t have to use the word LOCAL, but I think
>>> that’s anyway an improvement, personally.
>>>
>>> In future we will hopefully offer GLOBAL indexes, and IMO it is better
>>> to reify the distinction in the syntax.
>>>
>>> On 12 May 2023, at 17:29, Caleb Rackliffe 
>>> wrote:
>>>
>>> 
>>> We don't need to know everything about SAI's performance profile to plan
>>> and execute some small, reasonable things now for 5.0. I'm going to try to
>>> summarize the least controversial package of ideas from the discussion
>>> above. I've left out creating any new syntax. For example, I think CREATE
>>> LOCAL INDEX, while explicit, is just not necessary. We don't use CREATE
>>> LOCAL TABLE, although it has the same locality as our indexes.
>>>
>>> Okay, so the proposal for 5.0...
>>>
>>> 1.) Add a YAML option that specifies a default implementation for CREATE
>>> INDEX, and make this the legacy 2i for now. No existing DDL breaks. We
>>> don't have to commit to the absolute superiority of SAI.
>>> 2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go
>>> to market using CREATE CUSTOM INDEX, which feels...not so polished.
>>> (The backend for this already exists w/ CREATE CUSTOM INDEX.)
>>> 3.) Leave in place but deprecate (client warnings could work?) CREATE
>>> CUSTOM INDEX. Support the syntax for the foreseeable future.
>>>
>>> Can we live w/ this?
>>>
>>> I don't think any information about SAI we could possibly acquire before
>>> a 5.0 release would affect the reasonableness of this much.
>>>
>>>
>>> On Fri, May 12, 2023 at 10:54 AM Benedict  wrote:
>>>
>>>> if we didn't have copious amounts of (not all public, I know, working
>>>> on it) evidence
>>>>
>>>>
>>>> If that’s the assumption on which this proposal is based, let’s discuss
>>>> the evidence base first, as given the fundamentally different way they work
>>>> (almost diametrically opposite), I would want to see a very high quality of
>>>> evidence to support the claim.
>>>>
>>>> I don’t think we can resolve this conversation effectively until this
>>>> question is settled.
>>>>
>>>> On 12 May 2023, at 16:19, Caleb Rackliffe 
>>>> wrote:
>>>>
>>>> 
>>>> > This creates huge headaches for everyone successfully using 2i today
>>>> though, and SAI *is not* guaranteed to perform as well or better - it has a
>>>> very dif

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

> Now, giving this thread, there is pushback for a config to allow default
impl to change… but there is 0 pushback for new syntax to make this
explicit…. So maybe we should [POLL] for what syntax people want?

I think the essential question is whether we want the concept of a default
index. If we do, we need to figure that out now. If we don't then a new
syntax that forces it becomes interesting.

Given it seems most DBs have a default index (see Postgres, etc.), I tend
to lean toward having one, but that's me...

On Fri, May 12, 2023 at 12:20 PM David Capwell  wrote:

> I really dislike the idea of the same CQL doing different things based upon
> a per-node configuration.
>
>
> I agree with Brandon that changing CQL behaviour like this based on node
> config is really not ideal.
>
>
> I am cool adding such a config, and also cool keeping CREATE INDEX
> disabled by default…. But would like to point out that we have many configs
> that impact CQL and they are almost always local configs…
>
> Is CREATE INDEX even allowed?  This is a per node config. Right now you
> can block globally, enable on a single instance, create the index for your
> users, then revert the config change on the instance….
>
> All guardrails that define what we can do are per node configs…
>
> Now, giving this thread, there is pushback for a config to allow default
> impl to change… but there is 0 pushback for new syntax to make this
> explicit…. So maybe we should [POLL] for what syntax people want?
>
> if we decide before the 5.0 release that we have enough information to
> change the default (#1), we can change it in a matter of minutes.
>
>
> I am strongly against this… SAI is new for 5.0 so should be disabled by
> default; else we disrespect the idea that new features are disabled by
> default.  I am cool with our docs recommending if we do find its better in
> most cases, but we should not change the default in the same reason it
> lands in.
>
> On May 12, 2023, at 10:10 AM, Caleb Rackliffe 
> wrote:
>
> I don't want to cut over for 5.0 either way. I was more contrasting a
> configurable cutover in 5.0 vs. a hard cutover later.
>
> On Fri, May 12, 2023 at 12:09 PM Benedict  wrote:
>
>> If the performance characteristics are as clear cut as you think, then
>> maybe it will be an easy decision once the evidence is available for
>> everyone to consider?
>>
>> If not, then we probably can’t do the hard cutover and so the answer is
>> still pretty simple?
>>
>> On 12 May 2023, at 18:04, Caleb Rackliffe 
>> wrote:
>>
>> 
>> I don't particularly like the YAML solution either, but absent that,
>> we're back to fighting about whether we introduce entirely new syntax or
>> hard cut over to SAI at some point.
>>
>> We already have per-node configuration in the YAML that determines
>> whether or not we can create a 2i at all, right?
>>
>> What if we just do #2 and #3 and punt on everything else?
>>
>> On Fri, May 12, 2023 at 11:56 AM Benedict  wrote:
>>
>>> A table is not a local concept at all, it has a global primary index -
>>> that’s the core idea of Cassandra.
>>>
>>> I agree with Brandon that changing CQL behaviour like this based on node
>>> config is really not ideal. New syntax is by far the simplest and safest
>>> solution to this IMO. It doesn’t have to use the word LOCAL, but I think
>>> that’s anyway an improvement, personally.
>>>
>>> In future we will hopefully offer GLOBAL indexes, and IMO it is better
>>> to reify the distinction in the syntax.
>>>
>>> On 12 May 2023, at 17:29, Caleb Rackliffe 
>>> wrote:
>>>
>>> 
>>> We don't need to know everything about SAI's performance profile to plan
>>> and execute some small, reasonable things now for 5.0. I'm going to try to
>>> summarize the least controversial package of ideas from the discussion
>>> above. I've left out creating any new syntax. For example, I think CREATE
>>> LOCAL INDEX, while explicit, is just not necessary. We don't use CREATE
>>> LOCAL TABLE, although it has the same locality as our indexes.
>>>
>>> Okay, so the proposal for 5.0...
>>>
>>> 1.) Add a YAML option that specifies a default implementation for CREATE
>>> INDEX, and make this the legacy 2i for now. No existing DDL breaks. We
>>> don't have to commit to the absolute superiority of SAI.
>>> 2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go
>>> to market using CREATE CUSTOM INDEX, which feels...not so pol

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

Even if we don't want to allow a default, we can keep the same CREATE INDEX
syntax in place, and have a guardrail forcing (or not) the selection of an
implementation, right? This would be no worse than the YAML option we
already have for enabling 2i creation as a whole.

On Fri, May 12, 2023 at 12:28 PM Benedict  wrote:

> I’m not convinced a default index makes any sense, no. The trade-offs in a
> distributed setting are much more pronounced.
>
> Indexes in a local-only RDBMS are much simpler affairs; the trade offs are
> much more subtle than here.
>
> On 12 May 2023, at 18:24, Caleb Rackliffe 
> wrote:
>
> 
> > Now, giving this thread, there is pushback for a config to allow
> default impl to change… but there is 0 pushback for new syntax to make this
> explicit…. So maybe we should [POLL] for what syntax people want?
>
> I think the essential question is whether we want the concept of a default
> index. If we do, we need to figure that out now. If we don't then a new
> syntax that forces it becomes interesting.
>
> Given it seems most DBs have a default index (see Postgres, etc.), I tend
> to lean toward having one, but that's me...
>
> On Fri, May 12, 2023 at 12:20 PM David Capwell  wrote:
>
>> I really dislike the idea of the same CQL doing different things based upon
>> a per-node configuration.
>>
>>
>> I agree with Brandon that changing CQL behaviour like this based on node
>> config is really not ideal.
>>
>>
>> I am cool adding such a config, and also cool keeping CREATE INDEX
>> disabled by default…. But would like to point out that we have many configs
>> that impact CQL and they are almost always local configs…
>>
>> Is CREATE INDEX even allowed?  This is a per node config. Right now you
>> can block globally, enable on a single instance, create the index for your
>> users, then revert the config change on the instance….
>>
>> All guardrails that define what we can do are per node configs…
>>
>> Now, giving this thread, there is pushback for a config to allow default
>> impl to change… but there is 0 pushback for new syntax to make this
>> explicit…. So maybe we should [POLL] for what syntax people want?
>>
>> if we decide before the 5.0 release that we have enough information to
>> change the default (#1), we can change it in a matter of minutes.
>>
>>
>> I am strongly against this… SAI is new for 5.0 so should be disabled by
>> default; else we disrespect the idea that new features are disabled by
>> default.  I am cool with our docs recommending if we do find its better in
>> most cases, but we should not change the default in the same reason it
>> lands in.
>>
>> On May 12, 2023, at 10:10 AM, Caleb Rackliffe 
>> wrote:
>>
>> I don't want to cut over for 5.0 either way. I was more contrasting a
>> configurable cutover in 5.0 vs. a hard cutover later.
>>
>> On Fri, May 12, 2023 at 12:09 PM Benedict  wrote:
>>
>>> If the performance characteristics are as clear cut as you think, then
>>> maybe it will be an easy decision once the evidence is available for
>>> everyone to consider?
>>>
>>> If not, then we probably can’t do the hard cutover and so the answer is
>>> still pretty simple?
>>>
>>> On 12 May 2023, at 18:04, Caleb Rackliffe 
>>> wrote:
>>>
>>> 
>>> I don't particularly like the YAML solution either, but absent that,
>>> we're back to fighting about whether we introduce entirely new syntax or
>>> hard cut over to SAI at some point.
>>>
>>> We already have per-node configuration in the YAML that determines
>>> whether or not we can create a 2i at all, right?
>>>
>>> What if we just do #2 and #3 and punt on everything else?
>>>
>>> On Fri, May 12, 2023 at 11:56 AM Benedict  wrote:
>>>
>>>> A table is not a local concept at all, it has a global primary index -
>>>> that’s the core idea of Cassandra.
>>>>
>>>> I agree with Brandon that changing CQL behaviour like this based on
>>>> node config is really not ideal. New syntax is by far the simplest and
>>>> safest solution to this IMO. It doesn’t have to use the word LOCAL, but I
>>>> think that’s anyway an improvement, personally.
>>>>
>>>> In future we will hopefully offer GLOBAL indexes, and IMO it is better
>>>> to reify the distinction in the syntax.
>>>>
>>>> On 12 May 2023, at 17:29, Caleb Rackliffe 
>>>> wrote:
>>>>
>>>> 

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

If at some point in the glorious future we have global indexes, I'm sure we
can add GLOBAL to the syntax...sry, working on an ugly poll...

On Fri, May 12, 2023 at 1:24 PM Benedict  wrote:

> If folk should be reading up on the index type, doesn’t that conflict with
> your support of a default?
>
> Should there be different global and local defaults, once we have global
> indexes, or should we always default to a local index? Or a global one?
>
> On 12 May 2023, at 18:39, Mick Semb Wever  wrote:
>
> 
>
>>
>> Given it seems most DBs have a default index (see Postgres, etc.), I tend
>> to lean toward having one, but that's me...
>>
>
>
> I'm for it too.  Would be nice to enforce the setting is globally uniform
> to avoid the per-node problem. Or add a keyspace option.
>
> For users replaying <5 DDLs this would just require they set the default
> index to 2i.
> This is not a headache, it's a one-off action that can be clearly
> expressed in NEWS.
> It acts as a deprecation warning too.
> This prevents new uneducated users from creating the unintended index, it
> supports existing users, and it does not present SAI as the battle-tested
>  default.
>
> Agree with the poll, there's a number of different PoVs here already.  I'm
> not fond of the LOCAL addition,  I appreciate what it informs, but it's
> just not important enough IMHO (folk should be reading up on the index
> type).
>
>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

[POLL] Centralize existing syntax or create new syntax?

1.) CREATE INDEX ... USING  WITH OPTIONS...
2.) CREATE LOCAL INDEX ... USING ... WITH OPTIONS...  (same as 1, but adds
LOCAL keyword for clarity and separation from future GLOBAL indexes)

(In both cases, we deprecate w/ client warnings CREATE CUSTOM INDEX)


[POLL] Should there be a default? (YES/NO)


[POLL] What do do with the default?

1.) Allow a default, and switch it to SAI (no configurables)
2.) Allow a default, and stay w/ the legacy 2i (no configurables)
3.) YAML config to override default index (legacy 2i remains the default)
4.) YAML config/guardrail to require index type selection (not required by
default)

On Fri, May 12, 2023 at 12:39 PM Mick Semb Wever  wrote:

>
>> Given it seems most DBs have a default index (see Postgres, etc.), I tend
>> to lean toward having one, but that's me...
>>
>
>
> I'm for it too.  Would be nice to enforce the setting is globally uniform
> to avoid the per-node problem. Or add a keyspace option.
>
> For users replaying <5 DDLs this would just require they set the default
> index to 2i.
> This is not a headache, it's a one-off action that can be clearly
> expressed in NEWS.
> It acts as a deprecation warning too.
> This prevents new uneducated users from creating the unintended index, it
> supports existing users, and it does not present SAI as the battle-tested
>  default.
>
> Agree with the poll, there's a number of different PoVs here already.  I'm
> not fond of the LOCAL addition,  I appreciate what it informs, but it's
> just not important enough IMHO (folk should be reading up on the index
> type).
>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

...and to clarify, answers should be what you'd like to see for 5.0
specifically

On Fri, May 12, 2023 at 1:36 PM Caleb Rackliffe 
wrote:

> [POLL] Centralize existing syntax or create new syntax?
>
> 1.) CREATE INDEX ... USING  WITH OPTIONS...
> 2.) CREATE LOCAL INDEX ... USING ... WITH OPTIONS...  (same as 1, but
> adds LOCAL keyword for clarity and separation from future GLOBAL indexes)
>
> (In both cases, we deprecate w/ client warnings CREATE CUSTOM INDEX)
>
>
> [POLL] Should there be a default? (YES/NO)
>
>
> [POLL] What do do with the default?
>
> 1.) Allow a default, and switch it to SAI (no configurables)
> 2.) Allow a default, and stay w/ the legacy 2i (no configurables)
> 3.) YAML config to override default index (legacy 2i remains the default)
> 4.) YAML config/guardrail to require index type selection (not required by
> default)
>
> On Fri, May 12, 2023 at 12:39 PM Mick Semb Wever  wrote:
>
>>
>>> Given it seems most DBs have a default index (see Postgres, etc.), I
>>> tend to lean toward having one, but that's me...
>>>
>>
>>
>> I'm for it too.  Would be nice to enforce the setting is
>> globally uniform to avoid the per-node problem. Or add a keyspace option.
>>
>> For users replaying <5 DDLs this would just require they set the default
>> index to 2i.
>> This is not a headache, it's a one-off action that can be clearly
>> expressed in NEWS.
>> It acts as a deprecation warning too.
>> This prevents new uneducated users from creating the unintended index,
>> it supports existing users, and it does not present SAI as the
>> battle-tested default.
>>
>> Agree with the poll, there's a number of different PoVs here already.
>> I'm not fond of the LOCAL addition,  I appreciate what it informs, but it's
>> just not important enough IMHO (folk should be reading up on the index
>> type).
>>
>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

I don’t think there’s going to be any real support for doing it in 5.0 anyway at this point.On May 12, 2023, at 1:48 PM, Benedict  wrote:Given we have no data in front of us to make a decision regarding switching defaults, I don’t think it is suitable to include that option in this poll. In fact, until we have sufficient data to discuss that I’m going to put a hard veto on that on technical grounds.On 12 May 2023, at 19:41, Caleb Rackliffe  wrote:...and to clarify, answers should be what you'd like to see for 5.0 specificallyOn Fri, May 12, 2023 at 1:36 PM Caleb Rackliffe <calebrackli...@gmail.com> wrote:[POLL] Centralize existing syntax or create new syntax?1.) CREATE INDEX ... USING  WITH OPTIONS...2.) CREATE LOCAL INDEX ... USING ... WITH OPTIONS...  (same as 1, but adds LOCAL keyword for clarity and separation from future GLOBAL indexes)(In both cases, we deprecate w/ client warnings CREATE CUSTOM INDEX)[POLL] Should there be a default? (YES/NO)[POLL] What do do with the default?1.) Allow a default, and switch it to SAI (no configurables)2.) Allow a default, and stay w/ the legacy 2i (no configurables)3.) YAML config to override default index (legacy 2i remains the default)4.) YAML config/guardrail to require index type selection (not required by default)On Fri, May 12, 2023 at 12:39 PM Mick Semb Wever <m...@apache.org> wrote:Given it seems most DBs have a default index (see Postgres, etc.), I tend to lean toward having one, but that's me... I'm for it too.  Would be nice to enforce the setting is globally uniform to avoid the per-node problem. Or add a keyspace option. For users replaying <5 DDLs this would just require they set the default index to 2i.This is not a headache, it's a one-off action that can be clearly expressed in NEWS.It acts as a deprecation warning too.This prevents new uneducated users from creating the unintended index, it supports existing users, and it does not present SAI as the battle-tested default.Agree with the poll, there's a number of different PoVs here already.  I'm not fond of the LOCAL addition,  I appreciate what it informs, but it's just not important enough IMHO (folk should be reading up on the index type).

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

I might as well weigh in...

[POLL] Centralize existing syntax or create new syntax?

1.) CREATE INDEX ... USING ... WITH OPTIONS...

(I think the more important protection for users WRT local indexes should
come in the form of a guardrail prohibiting scatter/gather queries against
them.)

[POLL] Should there be a default? (YES/NO)

Yes

[POLL] What do do with the default?

3.) and 4.) Allow both configuring a default and disabling the default
concept entirely. (Both of these are creation-time items, just like the
current guardrails we have around 2i creation and SASI disabling.)

(No defaults should change, but administrators can lock this down/change
the default index without much effort.)

On Mon, May 15, 2023 at 10:52 PM guo Maxwell  wrote:

> [POLL] Centralize existing syntax or create new syntax?
>
>
> 1.) CREATE INDEX ... USING  WITH OPTIONS...
>
> and I  think we should keep CREATE CUSTOM INDEX
>
> [POLL] Should there be a default? (YES/NO)
>
>
> of course  YES
>
> [POLL] What do do with the default?
>
>
> 4.) YAML config/guardrail to require index type selection (not required by
> default)
>
>
>
> Jonathan Ellis  于2023年5月16日周二 07:18写道：
>
>> On Fri, May 12, 2023 at 1:39 PM Caleb Rackliffe 
>> wrote:
>>
>>> [POLL] Centralize existing syntax or create new syntax?
>>>
>>
>> 1 (Existing)
>>
>> [POLL] Should there be a default? (YES/NO)
>>>
>>
>> YES
>>
>>
>>> [POLL] What do do with the default?
>>>
>>
>> 1 (Default SAI)
>>
>>
>
>
> --
> you are the apple of my eye !
>

Re: [DISCUSS] Feature branch version hygiene

[Caleb Rackliffe]

I like the NA approach for sub-tasks that roll up to a parent/epic ticket.
When that lands, it gets a real version, and any sub-task is assumed to
also have that version.

Not that it has to be called "NA", but there should be something to denote
"inherits from parent".

On Tue, May 16, 2023 at 3:04 PM Benedict  wrote:

> Copying my rely on the ticket…
>
>
> We have this discussion roughly once per major. If you look back through
> dev@ you'll find the last one a few years back.
>
> I don't recall NA ever being the approved approach, though. ".x" lines are
> target versions, whereas concrete versions are the ones a fix landed in.
> There's always ambiguity over the next release, as it's sort of both. But
> since there is no 5.0 version, only 5.0-alphaN, 5.0-betaN and 5.0.0,
> perhaps 5.0 is the correct label (and makes sense to me). I forget what we
> landed upon last time.
>
> Work that has actually landed should probably be labelled as 5.0-alpha1
>
> On 16 May 2023, at 21:02, J. D. Jordan  wrote:
>
> 
>
> Process question/discussion. Should tickets that are merged to CEP feature
> branches, like  https://issues.apache.org/jira/browse/CASSANDRA-18204, have
> a fixver of 5.0 on them After merging to the feature branch?
>
>
> For the SAI CEP which is also using the feature branch method the
> "reviewed and merged to feature branch" tickets seem to be given a version
> of NA.
>
>
> Not sure that's the best “waiting for cep to merge” version either?  But
> it seems better than putting 5.0 on them to me.
>
>
> Why I’m not keen on 5.0 is because if we cut the release today those
> tickets would not be there.
>
>
> What do other people think?  Is there a better version designation we can
> use?
>
>
> On a different project I have in the past made a “version number” in JIRA
> for each long running feature branch. Tickets merged to the feature branch
> got the epic ticket number as their version, and then it got updated to the
> “real” version when the feature branch was merged to trunk.
>
>
> -Jeremiah
>
>

Re: [DISCUSS] Feature branch version hygiene

[Caleb Rackliffe]

...but that and "What do we do with things that might be in 5.0 and might
not?" are different questions.

A version that denotes "next major release" until merged (at which point it
could be given a real version) could be helpful...

On Tue, May 16, 2023 at 3:28 PM Caleb Rackliffe 
wrote:

> I like the NA approach for sub-tasks that roll up to a parent/epic ticket.
> When that lands, it gets a real version, and any sub-task is assumed to
> also have that version.
>
> Not that it has to be called "NA", but there should be something to denote
> "inherits from parent".
>
> On Tue, May 16, 2023 at 3:04 PM Benedict  wrote:
>
>> Copying my rely on the ticket…
>>
>>
>> We have this discussion roughly once per major. If you look back through
>> dev@ you'll find the last one a few years back.
>>
>> I don't recall NA ever being the approved approach, though. ".x" lines
>> are target versions, whereas concrete versions are the ones a fix landed
>> in. There's always ambiguity over the next release, as it's sort of both.
>> But since there is no 5.0 version, only 5.0-alphaN, 5.0-betaN and 5.0.0,
>> perhaps 5.0 is the correct label (and makes sense to me). I forget what we
>> landed upon last time.
>>
>> Work that has actually landed should probably be labelled as 5.0-alpha1
>>
>> On 16 May 2023, at 21:02, J. D. Jordan  wrote:
>>
>> 
>>
>> Process question/discussion. Should tickets that are merged to CEP
>> feature branches, like
>> https://issues.apache.org/jira/browse/CASSANDRA-18204, have a fixver of
>> 5.0 on them After merging to the feature branch?
>>
>>
>> For the SAI CEP which is also using the feature branch method the
>> "reviewed and merged to feature branch" tickets seem to be given a version
>> of NA.
>>
>>
>> Not sure that's the best “waiting for cep to merge” version either?  But
>> it seems better than putting 5.0 on them to me.
>>
>>
>> Why I’m not keen on 5.0 is because if we cut the release today those
>> tickets would not be there.
>>
>>
>> What do other people think?  Is there a better version designation we can
>> use?
>>
>>
>> On a different project I have in the past made a “version number” in JIRA
>> for each long running feature branch. Tickets merged to the feature branch
>> got the epic ticket number as their version, and then it got updated to the
>> “real” version when the feature branch was merged to trunk.
>>
>>
>> -Jeremiah
>>
>>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

> 1. What's up with naming anything "legacy". Calling the current index
type "2i" seems perfectly fine with me. From what I've heard it can work
great for many users?

We can give the existing default secondary index any public-facing name we
like, but "2i" is too broad. It just stands for "secondary index", which is
obviously broad enough to cover anything. The use of "legacy" is
conversational, and it reflects the assertion that SAI should, when at
feature parity, be superior to the existing default 2i implementation for
any workload w/ partition-restricted queries. It will surely be possible to
construct a scenario where SAI's SSTable-attached design, combined with
global scatter/gather queries and a huge number of local/per-node SSTables,
causes it to perform worse than the existing default 2i, which is just an
inverted index implemented as a hidden table w/ search terms as partition
keys.

> 2. It should be possible to always specify the index type explicitly. In
other words, it should be possible to CREATE CUSTOM INDEX ... USING "2i"
(if it isn't already)

Yes. It should be possible to specify the type no matter what syntax we
use. However, if we started this project from scratch, I don't think we
would build CREATE CUSTOM INDEX in the first place.

> 2b) It should be possible to just say "SAI" or "SASIIndex", not the full
Java path.
> 3. It's a fair point that the "CUSTOM" word may make this sound a bit too
special... The simplest change IMO is to just make the CUSTOM work optional.

Agreed on both, and 2b (aliasing) is already supported for CREATE CUSTOM
INDEX. (It may be that we should move toward something like a
ServiceLoader-enabled set of named 2i's.)

> 4. Benedict's point that a YAML option is per node is a good one... For
example, you wouldn't want some nodes to create a 2i index and other nodes
a SAI index for the same index That said, how many other YAML options
can you think of that would create total chaos if different nodes actually
had different values for them? For example what if a guardrail allowed some
action on some nodes but not others?  Maybe what we need is a jira ticket
to enforce that certain sections of the config must not differ?

At some point, my guess is that TCM will give us the ability to have
consistent, cluster-wide metadata/configuration. Right now, we have quite a
few YAML options that control cluster-wide behavior including our
prohibition on creating experimental SASI indexes and our option to disable
2i creation. None of the options we've discussed should make it possible
for a single secondary index on a column of a table to have differing local
implementations.

> 6. MySQL allows the DBA to determine the default engine. This seems to
work well. If the user doesn't care, they don't care, if they do, they use
the explicit syntax.

Sounds like option #3 on the 3rd POLL.

On Wed, May 17, 2023 at 3:33 PM Henrik Ingo 
wrote:

> I have read the thread but chose to reply to the top message...
>
> I'm coming to this with the background of having worked with MySQL, where
> both the storage engine and index implementation had many options, and
> often of course some index types were only available in some engines.
>
> I would humbly suggest:
>
> 1. What's up with naming anything "legacy". Calling the current index type
> "2i" seems perfectly fine with me. From what I've heard it can work great
> for many users?
>
> 2. It should be possible to always specify the index type explicitly. In
> other words, it should be possible to CREATE CUSTOM INDEX ... USING "2i"
> (if it isn't already)
>
> 2b) It should be possible to just say "SAI" or "SASIIndex", not the full
> Java path.
>
> 3. It's a fair point that the "CUSTOM" word may make this sound a bit too
> special... The simplest change IMO is to just make the CUSTOM work optional.
>
> 4. Benedict's point that a YAML option is per node is a good one... For
> example, you wouldn't want some nodes to create a 2i index and other nodes
> a SAI index for the same index That said, how many other YAML options
> can you think of that would create total chaos if different nodes actually
> had different values for them? For example what if a guardrail allowed some
> action on some nodes but not others?  Maybe what we need is a jira ticket
> to enforce that certain sections of the config must not differ?
>
> 5. That said, the default index type could also be a property of the
> keyspace
>
> 6. MySQL allows the DBA to determine the default engine. This seems to
> work well. If the user doesn't care, they don't care, if they do, they us

Re: [DISCUSS] Feature branch version hygiene

[Caleb Rackliffe]

So when a CEP slips, do we have to create a 5.1-cep-N? Could we just have a
version that's "NextMajorRelease" or something like that? It should still
be pretty easy to bulk replace if we have something else to filter on, like
belonging to an epic?

On Wed, May 17, 2023 at 6:42 PM Mick Semb Wever  wrote:

>
>
> On Tue, 16 May 2023 at 13:02, J. D. Jordan 
> wrote:
>
>> Process question/discussion. Should tickets that are merged to CEP
>> feature branches, like
>> https://issues.apache.org/jira/browse/CASSANDRA-18204, have a fixver of
>> 5.0 on them After merging to the feature branch?
>>
>>
>> For the SAI CEP which is also using the feature branch method the
>> "reviewed and merged to feature branch" tickets seem to be given a version
>> of NA.
>>
>>
>> Not sure that's the best “waiting for cep to merge” version either?  But
>> it seems better than putting 5.0 on them to me.
>>
>>
>> Why I’m not keen on 5.0 is because if we cut the release today those
>> tickets would not be there.
>>
>>
>> What do other people think?  Is there a better version designation we can
>> use?
>>
>>
>> On a different project I have in the past made a “version number” in JIRA
>> for each long running feature branch. Tickets merged to the feature branch
>> got the epic ticket number as their version, and then it got updated to the
>> “real” version when the feature branch was merged to trunk.
>>
>
>
> Thanks for raising the thread, I remember there was some confusion early
> wrt features branches too.
>
> To rehash, for everything currently resolved in trunk 5.0 is the correct
> fixVersion.  (And there should be no unresolved issues today with 5.0
> fixVersion, they should be 5.x)
>
>
> When alpha1 is cut, then the 5.0-alpha1 fixVersion is created and
> everything with 5.0 also gets  5.0-alpha1. At the same time 5.0-alpha2,
> 5.0-beta, 5.0-rc, 5.0.0 fixVersions are created. Here both 5.0-beta and
> 5.0-rc are blocking placeholder fixVersions: no resolved issues are left
> with this fixVersion the same as the .x placeholder fixVersions. The 5.0.0
> is also used as a blocking version, though it is also an eventual
> fixVersion for resolved tickets. Also note, all tickets up to and
> including 5.0.0 will also have the 5.0 fixVersion.
>
>
> A particular reason for doing things the way they are is to make it easy
> for the release manager to bulk correct fixVersions, at release time or
> even later, i.e. without having to read the ticket or go talk to authors
> or painstakingly crawl CHANGES.txt.
>
>
> For feature branches my suggestion is that we create a fixVersion for each
> of them, e.g. 5.0-cep-15
>
> Yup, that's your suggestion Jeremiah (I wrote this up on the plane before
> I got to read your post properly).
>
>
> (As you say) This then makes it easy to see where the code is (or what
> the patch is currently being based on). And when the feature branch is
> merged then it is easy to bulk replace it with trunk's fixVersion, e.g.  
> 5.0-cep-15
> with 5.0
>
>
> The NA fixVersion was introduced for the other repositories, e.g. website
> updates.
>

Re: [DISCUSS] Feature branch version hygiene

[Caleb Rackliffe]

...otherwise I'm fine w/ just the CEP name, like "CEP-7" for SAI, etc.

On Wed, May 17, 2023 at 11:24 PM Caleb Rackliffe 
wrote:

> So when a CEP slips, do we have to create a 5.1-cep-N? Could we just have
> a version that's "NextMajorRelease" or something like that? It should still
> be pretty easy to bulk replace if we have something else to filter on, like
> belonging to an epic?
>
> On Wed, May 17, 2023 at 6:42 PM Mick Semb Wever  wrote:
>
>>
>>
>> On Tue, 16 May 2023 at 13:02, J. D. Jordan 
>> wrote:
>>
>>> Process question/discussion. Should tickets that are merged to CEP
>>> feature branches, like
>>> https://issues.apache.org/jira/browse/CASSANDRA-18204, have a fixver of
>>> 5.0 on them After merging to the feature branch?
>>>
>>>
>>> For the SAI CEP which is also using the feature branch method the
>>> "reviewed and merged to feature branch" tickets seem to be given a version
>>> of NA.
>>>
>>>
>>> Not sure that's the best “waiting for cep to merge” version either?  But
>>> it seems better than putting 5.0 on them to me.
>>>
>>>
>>> Why I’m not keen on 5.0 is because if we cut the release today those
>>> tickets would not be there.
>>>
>>>
>>> What do other people think?  Is there a better version designation we
>>> can use?
>>>
>>>
>>> On a different project I have in the past made a “version number” in
>>> JIRA for each long running feature branch. Tickets merged to the feature
>>> branch got the epic ticket number as their version, and then it got updated
>>> to the “real” version when the feature branch was merged to trunk.
>>>
>>
>>
>> Thanks for raising the thread, I remember there was some confusion early
>> wrt features branches too.
>>
>> To rehash, for everything currently resolved in trunk 5.0 is the correct
>> fixVersion.  (And there should be no unresolved issues today with 5.0
>> fixVersion, they should be 5.x)
>>
>>
>> When alpha1 is cut, then the 5.0-alpha1 fixVersion is created and
>> everything with 5.0 also gets  5.0-alpha1. At the same time 5.0-alpha2,
>> 5.0-beta, 5.0-rc, 5.0.0 fixVersions are created. Here both 5.0-beta and
>> 5.0-rc are blocking placeholder fixVersions: no resolved issues are left
>> with this fixVersion the same as the .x placeholder fixVersions. The 5.0.0
>> is also used as a blocking version, though it is also an eventual
>> fixVersion for resolved tickets. Also note, all tickets up to and
>> including 5.0.0 will also have the 5.0 fixVersion.
>>
>>
>> A particular reason for doing things the way they are is to make it easy
>> for the release manager to bulk correct fixVersions, at release time or
>> even later, i.e. without having to read the ticket or go talk to authors
>> or painstakingly crawl CHANGES.txt.
>>
>>
>> For feature branches my suggestion is that we create a fixVersion for
>> each of them, e.g. 5.0-cep-15
>>
>> Yup, that's your suggestion Jeremiah (I wrote this up on the plane before
>> I got to read your post properly).
>>
>>
>> (As you say) This then makes it easy to see where the code is (or what
>> the patch is currently being based on). And when the feature branch is
>> merged then it is easy to bulk replace it with trunk's fixVersion, e.g.  
>> 5.0-cep-15
>> with 5.0
>>
>>
>> The NA fixVersion was introduced for the other repositories, e.g. website
>> updates.
>>
>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

Posted on ASF Slack to see if we can get more responses, but so far the
leaders seem to be...

[POLL] Centralize existing syntax or create new syntax?

1.) CREATE INDEX ... USING ... WITH OPTIONS...

(i.e. centralize)

[POLL] Should there be a default? (YES/NO)

Yes

[POLL] What do do with the default?

3.) and 4.) i.e. YAML options to control default and requirement to specify
a default

(i.e. w/o changing default in 5.0)

On Thu, May 18, 2023 at 3:33 AM Miklosovic, Stefan <
stefan.mikloso...@netapp.com> wrote:

> I don't want to hijack this thread, I just want to say that the point 4)
> seems to be recurring. I second Caleb in saying that transactional metadata
> would probably fix this. Because of the problem of not being sure that all
> config is same, cluster-wide, I basically dropped the effort on CEP-24
> because different local configurations might compromise the security.
>
> 
> From: Henrik Ingo 
> Sent: Wednesday, May 17, 2023 22:32
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] The future of CREATE INDEX
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I have read the thread but chose to reply to the top message...
>
> I'm coming to this with the background of having worked with MySQL, where
> both the storage engine and index implementation had many options, and
> often of course some index types were only available in some engines.
>
> I would humbly suggest:
>
> 1. What's up with naming anything "legacy". Calling the current index type
> "2i" seems perfectly fine with me. From what I've heard it can work great
> for many users?
>
> 2. It should be possible to always specify the index type explicitly. In
> other words, it should be possible to CREATE CUSTOM INDEX ... USING "2i"
> (if it isn't already)
>
> 2b) It should be possible to just say "SAI" or "SASIIndex", not the full
> Java path.
>
> 3. It's a fair point that the "CUSTOM" word may make this sound a bit too
> special... The simplest change IMO is to just make the CUSTOM work optional.
>
> 4. Benedict's point that a YAML option is per node is a good one... For
> example, you wouldn't want some nodes to create a 2i index and other nodes
> a SAI index for the same index That said, how many other YAML options
> can you think of that would create total chaos if different nodes actually
> had different values for them? For example what if a guardrail allowed some
> action on some nodes but not others?  Maybe what we need is a jira ticket
> to enforce that certain sections of the config must not differ?
>
> 5. That said, the default index type could also be a property of the
> keyspace
>
> 6. MySQL allows the DBA to determine the default engine. This seems to
> work well. If the user doesn't care, they don't care, if they do, they use
> the explicit syntax.
>
> henrik
>
>
> On Wed, May 10, 2023 at 12:45 AM Caleb Rackliffe  <mailto:calebrackli...@gmail.com>> wrote:
> Earlier today, Mick started a thread on the future of our index creation
> DDL on Slack:
>
> https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019<
> https://urldefense.com/v3/__https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019__;!!PbtH5S7Ebw!YuQzuQkxC0gmD9ofXEGoaEmVMwPwZ_ab8-B_PCfRfNsQtKIZDLOIuw38jnV1Vt8TqHXn-818hL-CoLbVJXBTCWgSxoE$
> >
>
> At the moment, there are two ways to create a secondary index.
>
> 1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()
>
> This creates an optionally named legacy 2i on the provided table and
> column.
>
> ex. CREATE INDEX my_index ON kd.tbl(my_text_col)
>
> 2.) CREATE CUSTOM INDEX [IF NOT EXISTS] [name] ON  () USING
>  [WITH OPTIONS = ]
>
> This creates a secondary index on the provided table and column using the
> specified 2i implementation class and (optional) parameters.
>
> ex. CREATE CUSTOM INDEX my_index ON ks.tbl(my_text_col) USING
> 'StorageAttachedIndex'
>
> (Note that the work on SAI added aliasing, so `StorageAttachedIndex` is
> shorthand for the fully-qualified class name, which is also valid.)
>
> So what is there to discuss?
>
> The concern Mick raised is...
>
> "...just folk continuing to use CREATE INDEX  because they think CREATE
> CUSTOM INDEX is advanced (or just don't know of it), and we leave users
> doing 2i (when they think they are, and/or we definitely want them to be,
> using SAI)"
>
> To paraphrase, we want people to use SAI once it's available where
> possible, and 

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[Caleb Rackliffe]

> Submodules do have their own overhead and edge cases, so I am mostly in
favor of using for cases where the code must live outside of tree (such as
jvm-dtest that lives out of tree as all branches need the same interfaces)

Agreed. Basically where I've ended up on this topic.

> We could go over some interesting examples such as testing 2i (SAI)

+100


On Wed, May 24, 2023 at 1:40 PM Alex Petrov  wrote:

> > I'm about to need to harry test for the paging across tombstone work for
> https://issues.apache.org/jira/browse/CASSANDRA-18424 (that's where my
> own overlapping fuzzing came in). In the process, I'll see if I can't
> distill something really simple along the lines of how React approaches it (
> https://react.dev/learn).
>
> We can pick that up as an example, sure.
>
> On Wed, May 24, 2023, at 4:53 PM, Josh McKenzie wrote:
>
> I have submitted a proposal to Cassandra Summit for a 4-hour Harry
> workshop,
>
> I'm about to need to harry test for the paging across tombstone work for
> https://issues.apache.org/jira/browse/CASSANDRA-18424 (that's where my
> own overlapping fuzzing came in). In the process, I'll see if I can't
> distill something really simple along the lines of how React approaches it (
> https://react.dev/learn).
>
> Ideally we'd be able to get something together that's a high level "In the
> next 15 minutes, you will know and understand A-G and have access to N% of
> the power of harry" kind of offer.
>
> Honestly, there's a *lot* in our ecosystem where we could benefit from
> taking a page from their book in terms of onboarding and getting started
> IMO.
>
> On Wed, May 24, 2023, at 10:31 AM, Alex Petrov wrote:
>
> > I wonder if a mini-onboarding session would be good as a community
> session - go over Harry, how to run it, how to add a test?  Would that be
> the right venue?  I just would like to see how we can not only plug it in
> to regular CI but get everyone that wants to add a test be able to know how
> to get started with it.
>
> I have submitted a proposal to Cassandra Summit for a 4-hour Harry
> workshop, but unfortunately it got declined. Goes without saying, we can
> still do it online, time and resources permitting. But again, I do not
> think it should be barring us from making Harry a part of the codebase, as
> it already is. In fact, we can be iterating on the development quicker
> having it in-tree.
>
> We could go over some interesting examples such as testing 2i (SAI),
> modelling Group By tests, or testing repair. If there is enough appetite
> and collaboration in the community, I will see if we can pull something
> like that together. Input on _what_ you would like to see / hear / tested
> is also appreciated. Harry was developed out of a strong need for
> large-scale testing, which also has informed many of its APIs, but we can
> make it easier to access for interactive testing / unit tests. We have been
> doing a lot of that with Transactional Metadata, too.
>
> > I'll hold off on this until Alex Petrov chimes in. @Alex -> got any
> thoughts here?
>
> Yes, sorry for not responding on this thread earlier. I can not understate
> how excited I am about this, and how important I think this is. Time
> constraints are somehow hard to overcome, but I hope the results brought by
> TCM will make it all worth it.
>
> On Wed, May 24, 2023, at 4:23 PM, Alex Petrov wrote:
>
> I think pulling Harry into the tree will make adoption easier for the
> folks. I have been a bit swamped with Transactional Metadata work, but I
> wanted to make some of the things we were using for testing TCM available
> outside of TCM branch. This includes a bunch of helper methods to perform
> operations on the clusters, data generation, and more useful stuff. Of
> course, the question always remains about how much time I want to spend
> porting it all to Gossip, but I think we can find a reasonable compromise.
>
> I would not set this improvement as a prerequisite to pulling Harry into
> the main branch, but rather interpret it as a commitment from myself to
> take community input and make it more approachable by the day.
>
> On Wed, May 24, 2023, at 2:44 PM, Josh McKenzie wrote:
>
> importantly it’s a million times better than the dtest-api process - which
> stymies development due to the friction.
>
> This is my major concern.
>
> What prompted this thread was harry being external to the core codebase
> and the lack of adoption and usage of it having led to atrophy of certain
> aspects of it, which then led to redundant implementation of some fuzz
> testing and lost time.
>
> We'd all be better served to have this closer to the main codebase as a
> forcing function to smooth out the rough edges, integrate it, and make it a
> collective artifact and first class citizen IMO.
>
> I have similar opinions about the dtest-api.
>
>
> On Wed, May 24, 2023, at 4:05 AM, Benedict wrote:
>
>
> It’s not without hiccups, and I’m sure we have more to learn. But it
> mostly just works, and importantly it’s a m

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[Caleb Rackliffe]

Isn’t the other reason Accord works well as a submodule that it has no dependencies on C* proper? Harry does at the moment, right? (Not that we couldn’t address that…just trying to think this through…)On May 24, 2023, at 6:54 PM, Benedict  wrote:In this case Harry is a testing module - it’s not something we will develop in tandem with C* releases, and we will want improvements to be applied across all branches.So it seems a natural fit for submodules to me.On 24 May 2023, at 21:09, Caleb Rackliffe  wrote:> Submodules do have their own overhead and edge cases, so I am mostly in favor of using for cases where the code must live outside of tree (such as jvm-dtest that lives out of tree as all branches need the same interfaces)Agreed. Basically where I've ended up on this topic.> We could go over some interesting examples such as testing 2i (SAI)+100On Wed, May 24, 2023 at 1:40 PM Alex Petrov <al...@coffeenco.de> wrote:> I'm about to need to harry test for the paging across tombstone work for https://issues.apache.org/jira/browse/CASSANDRA-18424 (that's where my own overlapping fuzzing came in). In the process, I'll see if I can't distill something really simple along the lines of how React approaches it (https://react.dev/learn).We can pick that up as an example, sure. On Wed, May 24, 2023, at 4:53 PM, Josh McKenzie wrote:I have submitted a proposal to Cassandra Summit for a 4-hour Harry workshop,I'm about to need to harry test for the paging across tombstone work for https://issues.apache.org/jira/browse/CASSANDRA-18424 (that's where my own overlapping fuzzing came in). In the process, I'll see if I can't distill something really simple along the lines of how React approaches it (https://react.dev/learn).Ideally we'd be able to get something together that's a high level "In the next 15 minutes, you will know and understand A-G and have access to N% of the power of harry" kind of offer.Honestly, there's a lot in our ecosystem where we could benefit from taking a page from their book in terms of onboarding and getting started IMO.On Wed, May 24, 2023, at 10:31 AM, Alex Petrov wrote:> I wonder if a mini-onboarding session would be good as a community session - go over Harry, how to run it, how to add a test?  Would that be the right venue?  I just would like to see how we can not only plug it in to regular CI but get everyone that wants to add a test be able to know how to get started with it.I have submitted a proposal to Cassandra Summit for a 4-hour Harry workshop, but unfortunately it got declined. Goes without saying, we can still do it online, time and resources permitting. But again, I do not think it should be barring us from making Harry a part of the codebase, as it already is. In fact, we can be iterating on the development quicker having it in-tree. We could go over some interesting examples such as testing 2i (SAI), modelling Group By tests, or testing repair. If there is enough appetite and collaboration in the community, I will see if we can pull something like that together. Input on _what_ you would like to see / hear / tested is also appreciated. Harry was developed out of a strong need for large-scale testing, which also has informed many of its APIs, but we can make it easier to access for interactive testing / unit tests. We have been doing a lot of that with Transactional Metadata, too. > I'll hold off on this until Alex Petrov chimes in. @Alex -> got any thoughts here?Yes, sorry for not responding on this thread earlier. I can not understate how excited I am about this, and how important I think this is. Time constraints are somehow hard to overcome, but I hope the results brought by TCM will make it all worth it.On Wed, May 24, 2023, at 4:23 PM, Alex Petrov wrote:I think pulling Harry into the tree will make adoption easier for the folks. I have been a bit swamped with Transactional Metadata work, but I wanted to make some of the things we were using for testing TCM available outside of TCM branch. This includes a bunch of helper methods to perform operations on the clusters, data generation, and more useful stuff. Of course, the question always remains about how much time I want to spend porting it all to Gossip, but I think we can find a reasonable compromise. I would not set this improvement as a prerequisite to pulling Harry into the main branch, but rather interpret it as a commitment from myself to take community input and make it more approachable by the day. On Wed, May 24, 2023, at 2:44 PM, Josh McKenzie wrote:importantly it’s a million times better than the dtest-api process - which stymies development due to the friction.This is my major concern.What prompted this thread was harry being external to the core codebase and the lack of adoption and usage of it having led to atrophy of certain aspects of it, which then led to redundant implementation of some fuzz testing and lo

Re: [VOTE] CEP-8 Datastax Drivers Donation

[Caleb Rackliffe]

+1

On Tue, Jun 13, 2023 at 11:25 AM Francisco Guerrero 
wrote:

> +1 (nb)
>
> On 2023/06/13 16:22:55 Andrés de la Peña wrote:
> > +1
> >
> > On Tue, 13 Jun 2023 at 16:40, Yifan Cai  wrote:
> >
> > > +1
> > > --
> > > *From:* David Capwell 
> > > *Sent:* Tuesday, June 13, 2023 8:37:10 AM
> > > *To:* dev 
> > > *Subject:* Re: [VOTE] CEP-8 Datastax Drivers Donation
> > >
> > > +1
> > >
> > > On Jun 13, 2023, at 7:59 AM, Josh McKenzie 
> wrote:
> > >
> > > +1
> > >
> > > On Tue, Jun 13, 2023, at 10:55 AM, Jeremiah Jordan wrote:
> > >
> > > +1 nb
> > >
> > > On Jun 13, 2023 at 9:14:35 AM, Jeremy Hanna <
> jeremy.hanna1...@gmail.com>
> > > wrote:
> > >
> > >
> > > Calling for a vote on CEP-8 [1].
> > >
> > > To clarify the intent, as Benjamin said in the discussion thread [2],
> the
> > > goal of this vote is simply to ensure that the community is in favor of
> > > the donation. Nothing more.
> > > The plan is to introduce the drivers, one by one. Each driver donation
> > > will need to be accepted first by the PMC members, as it is the case
> for
> > > any donation. Therefore the PMC should have full control on the pace at
> > > which new drivers are accepted.
> > >
> > > If this vote passes, we can start this process for the Java driver
> under
> > > the direction of the PMC.
> > >
> > > Jeremy
> > >
> > > 1.
> > >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation
> > > 2. https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp
> > >
> > >
> > >
> >
>

Re: [DISCUSS] The future of CREATE INDEX

[Caleb Rackliffe]

For everyone previously following this, just created
https://issues.apache.org/jira/browse/CASSANDRA-18615 :)

On Fri, May 19, 2023 at 1:34 PM Caleb Rackliffe 
wrote:

> Posted on ASF Slack to see if we can get more responses, but so far the
> leaders seem to be...
>
> [POLL] Centralize existing syntax or create new syntax?
>
> 1.) CREATE INDEX ... USING ... WITH OPTIONS...
>
> (i.e. centralize)
>
> [POLL] Should there be a default? (YES/NO)
>
> Yes
>
> [POLL] What do do with the default?
>
> 3.) and 4.) i.e. YAML options to control default and requirement to
> specify a default
>
> (i.e. w/o changing default in 5.0)
>
> On Thu, May 18, 2023 at 3:33 AM Miklosovic, Stefan <
> stefan.mikloso...@netapp.com> wrote:
>
>> I don't want to hijack this thread, I just want to say that the point 4)
>> seems to be recurring. I second Caleb in saying that transactional metadata
>> would probably fix this. Because of the problem of not being sure that all
>> config is same, cluster-wide, I basically dropped the effort on CEP-24
>> because different local configurations might compromise the security.
>>
>> 
>> From: Henrik Ingo 
>> Sent: Wednesday, May 17, 2023 22:32
>> To: dev@cassandra.apache.org
>> Subject: Re: [DISCUSS] The future of CREATE INDEX
>>
>> NetApp Security WARNING: This is an external email. Do not click links or
>> open attachments unless you recognize the sender and know the content is
>> safe.
>>
>>
>>
>> I have read the thread but chose to reply to the top message...
>>
>> I'm coming to this with the background of having worked with MySQL, where
>> both the storage engine and index implementation had many options, and
>> often of course some index types were only available in some engines.
>>
>> I would humbly suggest:
>>
>> 1. What's up with naming anything "legacy". Calling the current index
>> type "2i" seems perfectly fine with me. From what I've heard it can work
>> great for many users?
>>
>> 2. It should be possible to always specify the index type explicitly. In
>> other words, it should be possible to CREATE CUSTOM INDEX ... USING "2i"
>> (if it isn't already)
>>
>> 2b) It should be possible to just say "SAI" or "SASIIndex", not the full
>> Java path.
>>
>> 3. It's a fair point that the "CUSTOM" word may make this sound a bit too
>> special... The simplest change IMO is to just make the CUSTOM work optional.
>>
>> 4. Benedict's point that a YAML option is per node is a good one... For
>> example, you wouldn't want some nodes to create a 2i index and other nodes
>> a SAI index for the same index That said, how many other YAML options
>> can you think of that would create total chaos if different nodes actually
>> had different values for them? For example what if a guardrail allowed some
>> action on some nodes but not others?  Maybe what we need is a jira ticket
>> to enforce that certain sections of the config must not differ?
>>
>> 5. That said, the default index type could also be a property of the
>> keyspace
>>
>> 6. MySQL allows the DBA to determine the default engine. This seems to
>> work well. If the user doesn't care, they don't care, if they do, they use
>> the explicit syntax.
>>
>> henrik
>>
>>
>> On Wed, May 10, 2023 at 12:45 AM Caleb Rackliffe <
>> calebrackli...@gmail.com<mailto:calebrackli...@gmail.com>> wrote:
>> Earlier today, Mick started a thread on the future of our index creation
>> DDL on Slack:
>>
>> https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019<
>> https://urldefense.com/v3/__https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019__;!!PbtH5S7Ebw!YuQzuQkxC0gmD9ofXEGoaEmVMwPwZ_ab8-B_PCfRfNsQtKIZDLOIuw38jnV1Vt8TqHXn-818hL-CoLbVJXBTCWgSxoE$
>> >
>>
>> At the moment, there are two ways to create a secondary index.
>>
>> 1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()
>>
>> This creates an optionally named legacy 2i on the provided table and
>> column.
>>
>> ex. CREATE INDEX my_index ON kd.tbl(my_text_col)
>>
>> 2.) CREATE CUSTOM INDEX [IF NOT EXISTS] [name] ON  ()
>> USING  [WITH OPTIONS = ]
>>
>> This creates a secondary index on the provided table and column using the
>> specified 2i implementation class and (optional) parameters.
>>
>> ex. CREATE CUSTOM INDEX my_index ON ks.tbl(my_text_col) USING
>&

Status Update on CEP-7 Storage Attached Indexes (SAI)

[Caleb Rackliffe]

Hello there!

After much toil, the first phase of CEP-7 is nearing completion (see
CASSANDRA-16052 ).
There are presently two issues to resolve before we'd like to merge the
cep-7-sai feature branch and all its goodness to trunk:

CASSANDRA-18670  -
Importer should build SSTable indexes successfully before making new
SSTables readable (in review)

CASSANDRA-18673  -
Reduce size of per-SSTable index components (in progress)

(We've been getting clean CircleCI runs for a while now, and have been
using the multiplexer to sniff out as much flakiness as possible up front.)

Once merged to trunk, the next steps are:

1.) Finish a Harry model that we can use to further fuzz test SAI before
5.0 releases (see CASSANDRA-18275
). We've done a fair
amount of fuzz/randomized testing at the component level, but I'd still
consider Harry (at least around single-partition query use-cases) a
critical item for us to have confidence before release.

2.) Start pursuing Phase 2 items as time and our needs allow. (see
CASSANDRA-18473 )

A reminder, SAI is a secondary index, and therefore is by definition an
opt-in feature, and has no explicit "feature flag". However, its
availability to users is still subject to the secondary_indexes_enabled
guardrail, which currently defaults to allowing creation.

Any thoughts, questions, or comments on the pre-merge plan here?

Re: Status Update on CEP-7 Storage Attached Indexes (SAI)

[Caleb Rackliffe]

Just a quick update...

With CASSANDRA-18670
<https://issues.apache.org/jira/browse/CASSANDRA-18670> complete,
and all remaining items in the category of performance optimizations and
further testing, the process of merging to trunk will likely start today,
beginning with a final rebase on the current trunk and J11 and J17 test
runs.

On Tue, Jul 18, 2023 at 3:47 PM Caleb Rackliffe 
wrote:

> Hello there!
>
> After much toil, the first phase of CEP-7 is nearing completion (see
> CASSANDRA-16052 <https://issues.apache.org/jira/browse/CASSANDRA-16052>).
> There are presently two issues to resolve before we'd like to merge the
> cep-7-sai feature branch and all its goodness to trunk:
>
> CASSANDRA-18670 <https://issues.apache.org/jira/browse/CASSANDRA-18670> -
> Importer should build SSTable indexes successfully before making new
> SSTables readable (in review)
>
> CASSANDRA-18673 <https://issues.apache.org/jira/browse/CASSANDRA-18673> -
> Reduce size of per-SSTable index components (in progress)
>
> (We've been getting clean CircleCI runs for a while now, and have been
> using the multiplexer to sniff out as much flakiness as possible up front.)
>
> Once merged to trunk, the next steps are:
>
> 1.) Finish a Harry model that we can use to further fuzz test SAI before
> 5.0 releases (see CASSANDRA-18275
> <https://issues.apache.org/jira/browse/CASSANDRA-18275>). We've done a
> fair amount of fuzz/randomized testing at the component level, but I'd
> still consider Harry (at least around single-partition query use-cases) a
> critical item for us to have confidence before release.
>
> 2.) Start pursuing Phase 2 items as time and our needs allow. (see
> CASSANDRA-18473 <https://issues.apache.org/jira/browse/CASSANDRA-18473>)
>
> A reminder, SAI is a secondary index, and therefore is by definition an
> opt-in feature, and has no explicit "feature flag". However, its
> availability to users is still subject to the secondary_indexes_enabled
> guardrail, which currently defaults to allowing creation.
>
> Any thoughts, questions, or comments on the pre-merge plan here?
>

Re: Status Update on CEP-7 Storage Attached Indexes (SAI)

[Caleb Rackliffe]

Alright, the cep-7-sai branch is now merged to trunk!

Now we move to addressing the most urgent items from "Phase 2" (
CASSANDRA-18473 <https://issues.apache.org/jira/browse/CASSANDRA-18473>)
before (and in the case of some testing after) the 5.0 freeze...

On Wed, Jul 26, 2023 at 6:07 AM Jeremy Hanna 
wrote:

> Thanks Caleb and Mike and Zhao and Andres and Piotr and everyone else
> involved with the SAI implementation!
>
> On Jul 25, 2023, at 3:01 PM, Caleb Rackliffe 
> wrote:
>
> 
> Just a quick update...
>
> With CASSANDRA-18670
> <https://issues.apache.org/jira/browse/CASSANDRA-18670> complete, and all
> remaining items in the category of performance optimizations and further
> testing, the process of merging to trunk will likely start today, beginning
> with a final rebase on the current trunk and J11 and J17 test runs.
>
> On Tue, Jul 18, 2023 at 3:47 PM Caleb Rackliffe 
> wrote:
>
>> Hello there!
>>
>> After much toil, the first phase of CEP-7 is nearing completion (see
>> CASSANDRA-16052 <https://issues.apache.org/jira/browse/CASSANDRA-16052>).
>> There are presently two issues to resolve before we'd like to merge the
>> cep-7-sai feature branch and all its goodness to trunk:
>>
>> CASSANDRA-18670 <https://issues.apache.org/jira/browse/CASSANDRA-18670>
>> - Importer should build SSTable indexes successfully before making new
>> SSTables readable (in review)
>>
>> CASSANDRA-18673 <https://issues.apache.org/jira/browse/CASSANDRA-18673>
>> - Reduce size of per-SSTable index components (in progress)
>>
>> (We've been getting clean CircleCI runs for a while now, and have been
>> using the multiplexer to sniff out as much flakiness as possible up front.)
>>
>> Once merged to trunk, the next steps are:
>>
>> 1.) Finish a Harry model that we can use to further fuzz test SAI before
>> 5.0 releases (see CASSANDRA-18275
>> <https://issues.apache.org/jira/browse/CASSANDRA-18275>). We've done a
>> fair amount of fuzz/randomized testing at the component level, but I'd
>> still consider Harry (at least around single-partition query use-cases) a
>> critical item for us to have confidence before release.
>>
>> 2.) Start pursuing Phase 2 items as time and our needs allow. (see
>> CASSANDRA-18473 <https://issues.apache.org/jira/browse/CASSANDRA-18473>)
>>
>> A reminder, SAI is a secondary index, and therefore is by definition an
>> opt-in feature, and has no explicit "feature flag". However, its
>> availability to users is still subject to the secondary_indexes_enabled
>> guardrail, which currently defaults to allowing creation.
>>
>> Any thoughts, questions, or comments on the pre-merge plan here?
>>
>

Re: Tokenization and SAI query syntax

[Caleb Rackliffe]

Here are some additional bits of prior art, if anyone finds them useful:


The Stratio Lucene Index -
https://github.com/Stratio/cassandra-lucene-index#examples

Stratio was the reason C* added the "expr" functionality. They embedded
something similar to ElasticSearch JSON, which probably isn't my favorite
choice, but it's there.


The ElasticSearch match query syntax -
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

Again, not my favorite. It's verbose, and probably too powerful for us.


ElasticSearch's documentation for the basic Lucene query syntax -
https://www.elastic.co/guide/en/elasticsearch/reference/8.9/query-dsl-query-string-query.html#query-string-syntax

One idea is to take the basic Lucene index, which it seems we already have
some support for, and feed it to "expr". This is nice for two reasons:

1.) People can just write Lucene queries if they already know how.
2.) No changes to the grammar.

Lucene has distinct concepts of filtering and querying, and this is kind of
the latter. I'm not sure how, for example, we would want "expr" to interact
w/ filters on other column indexes in vanilla CQL space...


On Mon, Jul 24, 2023 at 9:37 AM Josh McKenzie  wrote:

> `column CONTAINS term`. Contains is used by both Java and Python for
> substring searches, so at least some users will be surprised by term-based
> behavior.
>
> I wonder whether users are in their "programming language" headspace or in
> their "querying a database" headspace when interacting with CQL? i.e. this
> would only present confusion if we expected users to be thinking in the
> idioms of their respective programming languages. If they're thinking in
> terms of SQL, MATCHES would probably end up confusing them a bit since it
> doesn't match the general structure of the MATCH operator.
>
> That said, I also think CONTAINS loses something important that you allude
> to here Jonathan:
>
> with corresponding query-time tokenization and analysis.  This means that
> the query term is not always a substring of the original string!  Besides
> obvious transformations like lowercasing, you have things like
> PhoneticFilter available as well.
>
> So to me, neither MATCHES nor CONTAINS are particularly great candidates.
>
> So +1 to the "I don't actually hate it" sentiment on:
>
> column : term`. Inspired by Lucene’s syntax
>
>
> On Mon, Jul 24, 2023, at 8:35 AM, Benedict wrote:
>
>
> I have a strong preference not to use the name of an SQL operator, since
> it precludes us later providing the SQL standard operator to users.
>
> What about CONTAINS TOKEN term? Or CONTAINS TERM term?
>
>
> On 24 Jul 2023, at 13:34, Andrés de la Peña  wrote:
>
> 
> `column = term` is definitively problematic because it creates an
> ambiguity when the queried column belongs to the primary key. For some
> queries we wouldn't know whether the user wants a primary key query using
> regular equality or an index query using the analyzer.
>
> `term_matches(column, term)` seems quite clear and hard to misinterpret,
> but it's quite long to write and its implementation will be challenging
> since we would need a bunch of special casing around SelectStatement and
> functions.
>
> LIKE, MATCHES and CONTAINS could be a bit misleading since they seem to
> evoke different behaviours to what they would have.
>
> `column LIKE :term:` seems a bit redundant compared to just using `column
> : term`, and we are still introducing a new symbol.
>
> I think I like `column : term` the most, because it's brief, it's similar
> to the equivalent Lucene's syntax, and it doesn't seem to clash with other
> different meanings that I can think of.
>
> On Mon, 24 Jul 2023 at 13:13, Jonathan Ellis  wrote:
>
> Hi all,
>
> With phase 1 of SAI wrapping up, I’d like to start the ball rolling on
> aligning around phase 2 features.
>
> In particular, we need to nail down the syntax for doing non-exact string
> matches.  We have a proof of concept that includes full Lucene analyzer and
> filter functionality – just the text transformation pieces, none of the
> storage parts – which is the gold standard in this space.  For example, the
> StandardAnalyzer [1] lowercases all terms and removes stopwords (common
> words like “a”, “is”, “the” that are usually not useful to search
> against).  Lucene also has classes that offer stemming, special case
> handling for email, and many languages besides English [2].
>
> What syntax should we use to express “rows whose analyzed tokens match
> this search term?”
>
> The syntax must be clear that we want to look for this term within the
> column data using the configured index with corresponding query-time
> tokenization and analysis.  This means that the query term is not always a
> substring of the original string!  Besides obvious transformations like
> lowercasing, you have things like PhoneticFilter available as well.
>
> Here are my thoughts on some of the options:
>
> `column = term`.  This is what the POC does to

Re: Tokenization and SAI query syntax

[Caleb Rackliffe]

For what it's worth, I'd very much like to completely remove SASI from the
codebase for 6.0. The only remaining functionality gaps at the moment are
LIKE (prefix/suffix) queries and its limited tokenization
capabilities, both of which already have SAI Phase 2 Jiras.

On Wed, Aug 2, 2023 at 7:20 PM Jeremiah Jordan 
wrote:

> SASI just uses “=“ for the tokenized equality matching, which is the exact
> thing this discussion is about changing/not liking.
>
> > On Aug 2, 2023, at 7:18 PM, J. D. Jordan 
> wrote:
> >
> > I do not think LIKE actually applies here. LIKE is used for prefix,
> contains, or suffix searches in SASI depending on the index type.
> >
> > This is about exact matching of tokens.
> >
> >> On Aug 2, 2023, at 5:53 PM, Jon Haddad 
> wrote:
> >>
> >> Certain bits of functionality also already exist on the SASI side of
> things, but I'm not sure how much overlap there is.  Currently, there's a
> LIKE keyword that handles token matching, although it seems to have some
> differences from the feature set in SAI.
> >>
> >> That said, there seems to be enough of an overlap that it would make
> sense to consider using LIKE in the same manner, doesn't it?  I think it
> would be a little odd if we have different syntax for different indexes.
> >>
> >> https://github.com/apache/cassandra/blob/trunk/doc/SASI.md
> >>
> >> I think one complication here is that there seems to be a desire, that
> I very much agree with, to expose as much of the underlying flexibility of
> Lucene as much as possible.  If it means we use Caleb's suggestion, I'd ask
> that the queries that SASI and SAI both support use the same syntax, even
> if it means there's two ways of writing the same query.  To use Caleb's
> example, this would mean supporting both LIKE and the `expr` column.
> >>
> >> Jon
> >>
> >>>> On 2023/08/01 19:17:11 Caleb Rackliffe wrote:
> >>> Here are some additional bits of prior art, if anyone finds them
> useful:
> >>>
> >>>
> >>> The Stratio Lucene Index -
> >>> https://github.com/Stratio/cassandra-lucene-index#examples
> >>>
> >>> Stratio was the reason C* added the "expr" functionality. They embedded
> >>> something similar to ElasticSearch JSON, which probably isn't my
> favorite
> >>> choice, but it's there.
> >>>
> >>>
> >>> The ElasticSearch match query syntax -
> >>>
> https://urldefense.com/v3/__https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html__;!!PbtH5S7Ebw!ZHwYJ2xkivwTzYgjkp5QFAzALXCWPqkga6GBD-m2aK3j06ioSCRPsdZD0CIe50VpRrtW-1rY_m6lrSpp7zVlAf0MsxZ9$
> >>>
> >>> Again, not my favorite. It's verbose, and probably too powerful for us.
> >>>
> >>>
> >>> ElasticSearch's documentation for the basic Lucene query syntax -
> >>>
> https://urldefense.com/v3/__https://www.elastic.co/guide/en/elasticsearch/reference/8.9/query-dsl-query-string-query.html*query-string-syntax__;Iw!!PbtH5S7Ebw!ZHwYJ2xkivwTzYgjkp5QFAzALXCWPqkga6GBD-m2aK3j06ioSCRPsdZD0CIe50VpRrtW-1rY_m6lrSpp7zVlAXEPP1sK$
> >>>
> >>> One idea is to take the basic Lucene index, which it seems we already
> have
> >>> some support for, and feed it to "expr". This is nice for two reasons:
> >>>
> >>> 1.) People can just write Lucene queries if they already know how.
> >>> 2.) No changes to the grammar.
> >>>
> >>> Lucene has distinct concepts of filtering and querying, and this is
> kind of
> >>> the latter. I'm not sure how, for example, we would want "expr" to
> interact
> >>> w/ filters on other column indexes in vanilla CQL space...
> >>>
> >>>
> >>>> On Mon, Jul 24, 2023 at 9:37 AM Josh McKenzie 
> wrote:
> >>>>
> >>>> `column CONTAINS term`. Contains is used by both Java and Python for
> >>>> substring searches, so at least some users will be surprised by
> term-based
> >>>> behavior.
> >>>>
> >>>> I wonder whether users are in their "programming language" headspace
> or in
> >>>> their "querying a database" headspace when interacting with CQL? i.e.
> this
> >>>> would only present confusion if we expected users to be thinking in
> the
> >>>> idioms of their respective programming languages. If they're thinking
> in
> >>

Re: Tokenization and SAI query syntax

[Caleb Rackliffe]

@Benedict I'm not particularly keen to try to graft the Lucene syntax into
CQL itself, to be clear. What I'm proposing is more along the lines of
allowing that syntax via "expr" and leaving that Lucene systems would call
"filters" in predicates currently expressible by CQL.

On Mon, Aug 7, 2023 at 5:12 AM Benedict  wrote:

> I’m strongly opposed to :
>
> It is very dissimilar to our current operators. CQL is already not the
> prettiest language, but let’s not make it a total mish mash.
>
>
>
> On 7 Aug 2023, at 10:59, Mike Adamson  wrote:
>
> 
> I am also in agreement with 'column : token' in that 'I don't hate it' but
> I'd like to offer an alternative to this in 'column HAS token'. HAS is
> currently not a keyword that we use so wouldn't cause any brain conflicts.
>
> While I don't hate ':' I have a particular dislike of the lucene search
> syntax because of its terseness and lack of easy readability.
>
> Saying that, I'm happy to do with ':' if that is the decision.
>
> On Fri, 4 Aug 2023 at 00:23, Jon Haddad 
> wrote:
>
>> Assuming SAI is a superset of SASI, and we were to set up something so
>> that SASI indexes auto convert to SAI, this gives even more weight to my
>> point regarding how differing behavior for the same syntax can lead to
>> issues.  Imo the best case scenario results in the user not even noticing
>> their indexes have changed.
>>
>> An (maybe better?) alternative is to add a flag to the index
>> configuration for "compatibility mod", which might address the concerns
>> around using an equality operator when it actually is a partial match.
>>
>> For what it's worth, I'm in agreement that = should mean full equality
>> and not token match.
>>
>> On 2023/08/03 03:56:23 Caleb Rackliffe wrote:
>> > For what it's worth, I'd very much like to completely remove SASI from
>> the
>> > codebase for 6.0. The only remaining functionality gaps at the moment
>> are
>> > LIKE (prefix/suffix) queries and its limited tokenization
>> > capabilities, both of which already have SAI Phase 2 Jiras.
>> >
>> > On Wed, Aug 2, 2023 at 7:20 PM Jeremiah Jordan 
>> > wrote:
>> >
>> > > SASI just uses “=“ for the tokenized equality matching, which is the
>> exact
>> > > thing this discussion is about changing/not liking.
>> > >
>> > > > On Aug 2, 2023, at 7:18 PM, J. D. Jordan > >
>> > > wrote:
>> > > >
>> > > > I do not think LIKE actually applies here. LIKE is used for prefix,
>> > > contains, or suffix searches in SASI depending on the index type.
>> > > >
>> > > > This is about exact matching of tokens.
>> > > >
>> > > >> On Aug 2, 2023, at 5:53 PM, Jon Haddad > >
>> > > wrote:
>> > > >>
>> > > >> Certain bits of functionality also already exist on the SASI side
>> of
>> > > things, but I'm not sure how much overlap there is.  Currently,
>> there's a
>> > > LIKE keyword that handles token matching, although it seems to have
>> some
>> > > differences from the feature set in SAI.
>> > > >>
>> > > >> That said, there seems to be enough of an overlap that it would
>> make
>> > > sense to consider using LIKE in the same manner, doesn't it?  I think
>> it
>> > > would be a little odd if we have different syntax for different
>> indexes.
>> > > >>
>> > > >> https://github.com/apache/cassandra/blob/trunk/doc/SASI.md
>> > > >>
>> > > >> I think one complication here is that there seems to be a desire,
>> that
>> > > I very much agree with, to expose as much of the underlying
>> flexibility of
>> > > Lucene as much as possible.  If it means we use Caleb's suggestion,
>> I'd ask
>> > > that the queries that SASI and SAI both support use the same syntax,
>> even
>> > > if it means there's two ways of writing the same query.  To use
>> Caleb's
>> > > example, this would mean supporting both LIKE and the `expr` column.
>> > > >>
>> > > >> Jon
>> > > >>
>> > > >>>> On 2023/08/01 19:17:11 Caleb Rackliffe wrote:
>> > > >>> Here are some additional bits of prior art, if anyone finds them
>> > > use

Re: Tokenization and SAI query syntax

[Caleb Rackliffe]

> I do not think we should start using lucene syntax for it, it will make
people think they can do everything else lucene allows.

I'm sure we won't be supporting everything Lucene allows, but this is going
to evolve. Right off the bat, if you introduce support for tokenization and
filtering, someone is, for example, going to ask for phrase queries. ("John
Smith landed in Virginia" is tokenized, but someone wants to match exactly
on "John Smith".) The whole point of the Vector project is to do relevance,
right? Are we going to do term boosting? Do we need queries like "field:
quick brown +fox -news" where fox must be present, news cannot be present,
and quick and brown increase relevance?

SASI uses "=" and "LIKE" in a way that assumes the user understands the
tokenization scheme in use on the target field. I understand that's a bit
ambiguous.

If we object to allowing expr embedding of a subset of the Lucene syntax, I
can't imagine we're okay w/ then jamming a subset of that syntax into the
main CQL grammar.

If we want to do this in non-expr CQL space, I think using functions
(ignoring the implementation complexity) at least removes ambiguity.
"token_match", "phrase_match", "token_like", "=", and "LIKE" would all be
pretty clear, although there may be other problems. For instance, what
happens when I try to use "token_match" on an indexed field whose analyzer
does not tokenize? We obviously can't use the index, so we'd be reduced to
requiring a filtering query, but maybe that's fine. My point is that, if
we're going to make write and read analyzers symmetrical, there's really no
way to make the semantics of our queries totally independent of analysis.
(ex. "field : foo bar" behaves differently w/ read tokenization than it
does without. It could even be an OR or AND query w/ tokenization,
depending on our defaults.)

On Mon, Aug 7, 2023 at 12:55 PM Atri Sharma  wrote:

> Why not start with SQLish operators supported by many databases (LIKE and
> CONTAINS)?
>
> On Mon, Aug 7, 2023 at 10:01 PM J. D. Jordan 
> wrote:
>
>> I am also -1 on directly exposing lucene like syntax here. Besides being
>> ugly, SAI is not lucene, I do not think we should start using lucene syntax
>> for it, it will make people think they can do everything else lucene allows.
>>
>> On Aug 7, 2023, at 5:13 AM, Benedict  wrote:
>>
>> 
>> I’m strongly opposed to :
>>
>> It is very dissimilar to our current operators. CQL is already not the
>> prettiest language, but let’s not make it a total mish mash.
>>
>>
>>
>> On 7 Aug 2023, at 10:59, Mike Adamson  wrote:
>>
>> 
>> I am also in agreement with 'column : token' in that 'I don't hate it'
>> but I'd like to offer an alternative to this in 'column HAS token'. HAS is
>> currently not a keyword that we use so wouldn't cause any brain conflicts.
>>
>> While I don't hate ':' I have a particular dislike of the lucene search
>> syntax because of its terseness and lack of easy readability.
>>
>> Saying that, I'm happy to do with ':' if that is the decision.
>>
>> On Fri, 4 Aug 2023 at 00:23, Jon Haddad 
>> wrote:
>>
>>> Assuming SAI is a superset of SASI, and we were to set up something so
>>> that SASI indexes auto convert to SAI, this gives even more weight to my
>>> point regarding how differing behavior for the same syntax can lead to
>>> issues.  Imo the best case scenario results in the user not even noticing
>>> their indexes have changed.
>>>
>>> An (maybe better?) alternative is to add a flag to the index
>>> configuration for "compatibility mod", which might address the concerns
>>> around using an equality operator when it actually is a partial match.
>>>
>>> For what it's worth, I'm in agreement that = should mean full equality
>>> and not token match.
>>>
>>> On 2023/08/03 03:56:23 Caleb Rackliffe wrote:
>>> > For what it's worth, I'd very much like to completely remove SASI from
>>> the
>>> > codebase for 6.0. The only remaining functionality gaps at the moment
>>> are
>>> > LIKE (prefix/suffix) queries and its limited tokenization
>>> > capabilities, both of which already have SAI Phase 2 Jiras.
>>> >
>>> > On Wed, Aug 2, 2023 at 7:20 PM Jeremiah Jordan 
>>> > wrote:
>>> >
>>> > > SASI just uses “=“ for the tokenized equality matching, which is the
>>> exact

Re: [DISCUSS] CASSANDRA-18743 Deprecation of metrics-reporter-config

[Caleb Rackliffe]

+1

> On Aug 11, 2023, at 8:10 AM, Brandon Williams  wrote:
> 
> +1
> 
> Kind Regards,
> Brandon
> 
>> On Fri, Aug 11, 2023 at 8:08 AM Ekaterina Dimitrova
>>  wrote:
>> 
>> 
>> “ The rationale for this proposed deprecation is that the upcoming 5.0 
>> release is a good time to evaluate dependencies that are no longer receiving 
>> updates and will become risks in the future.”
>> 
>> Thank you for raising it, I support your proposal for deprecation
>> 
>>> On Fri, 11 Aug 2023 at 8:55, Abe Ratnofsky  wrote:
>>> 
>>> Hey folks,
>>> 
>>> Opening a thread to get input on a proposed dependency deprecation in 5.0: 
>>> metrics-reporter-config has been archived for 3 years and not updated in 
>>> nearly 6 years.
>>> 
>>> This project has a minor security issue with its usage of unsafe YAML 
>>> loading via snakeyaml’s unprotected Constructor: 
>>> https://nvd.nist.gov/vuln/detail/CVE-2022-1471
>>> 
>>> This CVE is reasonable to suppress, since operators should be able to trust 
>>> their YAML configuration files.
>>> 
>>> The rationale for this proposed deprecation is that the upcoming 5.0 
>>> release is a good time to evaluate dependencies that are no longer 
>>> receiving updates and will become risks in the future.
>>> 
>>> https://issues.apache.org/jira/browse/CASSANDRA-18743
>>> 
>>> —
>>> Abe
>>> 

Re: Tokenization and SAI query syntax

[Caleb Rackliffe]

We’ve already started down the path of using a git sub-module for the Accord 
library. That could be an option at some point.

> On Aug 13, 2023, at 12:53 PM, Jon Haddad  wrote:
> 
> Functions make sense to me too.  In addition to the reasons listed, I if we 
> acknowledge that functions in predicates are inevitable, then it makes total 
> sense to use them here.  I think this is the most forward thinking approach.
> 
> Assuming this happens, one thing that would be great down the line would be 
> if the CQL parser was broken out into a subproject with an artifact published 
> so the soon to be additional complexity of parsing CQL didn't have to be 
> pushed to every single end user like it does today.  I'm not trying to expand 
> the scope right now, just laying an idea down for the future.  
> 
> Jon
> 
>> On 2023/08/07 21:26:40 Josh McKenzie wrote:
>> Been chatting a bit w/Caleb about this offline and poking around to better 
>> educate myself.
>> 
>>> using functions (ignoring the implementation complexity) at least removes 
>>> ambiguity. 
>> This, plus using functions lets us kick the can down the road a bit in terms 
>> of landing on an integrated grammar we agree on. It seems to me there's a 
>> tension between:
>> 1. "SQL-like" (i.e. postgres-like)
>> 2. "Indexing and Search domain-specific-like" (i.e. lucene syntax which, as 
>> Benedict points out, doesn't really jell w/what we have in CQL at this 
>> point), and
>> 3. ??? Some other YOLO CQL / C* specific thing where we go our own road
>> I don't think we're really going to know what our feature-set in terms of 
>> indexing is going to look like or the shape it's going to take for awhile, 
>> so backing ourselves into any of the 3 corners above right now feels very 
>> premature to me.
>> 
>> So I'm coming around to the expr / method call approach to preserve that 
>> flexibility. It's maximally explicit and preserves optionality at the 
>> expense of being clunky. For now.
>> 
>> On Mon, Aug 7, 2023, at 4:00 PM, Caleb Rackliffe wrote:
>>>> I do not think we should start using lucene syntax for it, it will make 
>>>> people think they can do everything else lucene allows.
>>> 
>>> I'm sure we won't be supporting everything Lucene allows, but this is going 
>>> to evolve. Right off the bat, if you introduce support for tokenization and 
>>> filtering, someone is, for example, going to ask for phrase queries. ("John 
>>> Smith landed in Virginia" is tokenized, but someone wants to match exactly 
>>> on "John Smith".) The whole point of the Vector project is to do relevance, 
>>> right? Are we going to do term boosting? Do we need queries like "field: 
>>> quick brown +fox -news" where fox must be present, news cannot be present, 
>>> and quick and brown increase relevance?
>>> 
>>> SASI uses "=" and "LIKE" in a way that assumes the user understands the 
>>> tokenization scheme in use on the target field. I understand that's a bit 
>>> ambiguous.
>>> 
>>> If we object to allowing expr embedding of a subset of the Lucene syntax, I 
>>> can't imagine we're okay w/ then jamming a subset of that syntax into the 
>>> main CQL grammar.
>>> 
>>> If we want to do this in non-expr CQL space, I think using functions 
>>> (ignoring the implementation complexity) at least removes ambiguity. 
>>> "token_match", "phrase_match", "token_like", "=", and "LIKE" would all be 
>>> pretty clear, although there may be other problems. For instance, what 
>>> happens when I try to use "token_match" on an indexed field whose analyzer 
>>> does not tokenize? We obviously can't use the index, so we'd be reduced to 
>>> requiring a filtering query, but maybe that's fine. My point is that, if 
>>> we're going to make write and read analyzers symmetrical, there's really no 
>>> way to make the semantics of our queries totally independent of analysis. 
>>> (ex. "field : foo bar" behaves differently w/ read tokenization than it 
>>> does without. It could even be an OR or AND query w/ tokenization, 
>>> depending on our defaults.)
>>> 
>>> On Mon, Aug 7, 2023 at 12:55 PM Atri Sharma  wrote:
>>>> Why not start with SQLish operators supported by many databases (LIKE and 
>>>> CONTAINS)?
>>>> 
>>>> On Mo

Re: [DISCUSS] Update default disk_access_mode to mmap_index_only on 5.0

[Caleb Rackliffe]

+100 to this

We'd have to come up w/ a pretty compelling counterexample to NOT switch
the default to mmap_index_only at this point.

On Wed, Sep 6, 2023 at 11:40 AM Brandon Williams  wrote:

> Given https://issues.apache.org/jira/browse/CASSANDRA-17237 I think it
> makes sense.  At the least I think we should restore disk_access_mode
> so that users are more aware of the options available.
>
> Kind Regards,
> Brandon
>
> On Wed, Sep 6, 2023 at 10:50 AM Paulo Motta 
> wrote:
> >
> > Hi,
> >
> > I've been bitten by OOMs with disk_access_mode:auto/mmap that were fixed
> by changing to disk_access_mode:mmap_index_only. In a particular benchmark
> I got 5x more read throughput on 3.11.x with disk_access_mode:
> mmap_index_only vs disk_access_mode: auto/mmap.
> >
> > Changing disk_access_mode to mmap_index_only seems to be a common
> recommendation on forums[1][2][3][4] and slack (find by searching
> disk_access_mode in the #cassandra channel on https://the-asf.slack.com/).
> >
> > It's not clear to me when using the default disk_access_mode:auto/mmap
> is beneficial, perhaps only when the read set fits in memory? Mick seems to
> think on CASSANDRA-15531 [5], that mmap_index_only has a higher heap cost
> and should be only used when warranted. However it's not uncommon to see
> people being bitten with OOMs or lower read performance due to the default
> disk_access_mode, so it makes me think it's not the best fool-proof default.
> >
> > Should we consider changing default "auto" behavior of
> "disk_access_mode" to be "mmap_index_only" instead of "mmap" in 5.0 since
> it's likely safer and perhaps more performant?
> >
> > Thanks,
> >
> > Paulo
> >
> > [1]
> https://stackoverflow.com/questions/72272035/troubleshooting-and-fixing-cassandra-oom-issue
> > [2] https://phabricator.wikimedia.org/T137419
> > [3] https://stackoverflow.com/a/55975471
> > [4]
> https://support.datastax.com/s/article/FAQ-Use-of-disk-access-mode-in-DSE-51-and-earlier
> > [5] https://issues.apache.org/jira/browse/CASSANDRA-15531
>

Re: [DISCUSS] Backport CASSANDRA-18816 to 5.0? Add support for repair coordinator to retry messages that timeout

[Caleb Rackliffe]

+1 on a 5.0 backport

On Wed, Sep 20, 2023 at 2:26 PM Brandon Williams  wrote:

> I think it could be argued that not retrying messages is a bug, I am
> +1 on including this in 5.0.
>
> Kind Regards,
> Brandon
>
> On Tue, Sep 19, 2023 at 1:16 PM David Capwell  wrote:
> >
> > To try to get repair more stable, I added optional retry logic (patch is
> still in review) to a handful of critical repair verbs.  This patch is
> disabled by default but allows you to opt-in to retries so ephemeral issues
> don’t cause a repair to fail after running for a long time (assuming they
> resolve within the retry window). There are 2 protocol level changes to
> enable this: VALIDATION_RSP and SYNC_RSP now send an ACK (if the sender
> doesn’t attach a callback, these ACKs get ignored in all versions; see
> org.apache.cassandra.net.ResponseVerbHandler#doVerb and
> Verb.REPAIR_RSP).  Given that we have already forked, I believe we would
> need to give a waiver to allow this patch due to this change.
> >
> > The patch was written on trunk, but figured back porting 5.0 would be
> rather trivial and this was brought up during the review, so floating this
> to a wider audience.
> >
> > If you look at the patch you will see that it is very large, but this is
> only to make testing of repair coordination easier and deterministic, the
> biggest code changes are:
> >
> > 1) Moving from ActiveRepairService.instance to
> ActiveRepairService.instance() (this is the main reason so many files were
> touched; this was needed so unit tests don’t load the whole world)
> > 2) Repair no longer reaches into global space and instead is provided
> the subsystems needed to perform repair; this change is local to repair code
> >
> > Both of these changes were only for testing as they allow us to simulate
> 1k repairs in around 15 seconds with 100% deterministic execution.
>

Re: [VOTE] Accept java-driver

[Caleb Rackliffe]

+1

On Tue, Oct 3, 2023 at 2:49 PM Sylvain Lebresne  wrote:

> +1
> --
> Sylvain
>
>
> On Tue, Oct 3, 2023 at 8:43 PM Jon Haddad 
> wrote:
>
>> +1
>>
>>
>> On 2023/10/03 04:52:47 Mick Semb Wever wrote:
>> > The donation of the java-driver is ready for its IP Clearance vote.
>> > https://incubator.apache.org/ip-clearance/cassandra-java-driver.html
>> >
>> > The SGA has been sent to the ASF.  This does not require acknowledgement
>> > before the vote.
>> >
>> > Once the vote passes, and the SGA has been filed by the ASF Secretary,
>> we
>> > will request ASF Infra to move the datastax/java-driver as-is to
>> > apache/java-driver
>> >
>> > This means all branches and tags, with all their history, will be
>> kept.  A
>> > cleaning effort has already cleaned up anything deemed not needed.
>> >
>> > Background for the donation is found in CEP-8:
>> >
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+DataStax+Drivers+Donation
>> >
>> > PMC members, please take note of (and check) the IP Clearance
>> requirements
>> > when voting.
>> >
>> > The vote will be open for 72 hours (or longer). Votes by PMC members are
>> > considered binding. A vote passes if there are at least three binding
>> +1s
>> > and no -1's.
>> >
>> > regards,
>> > Mick
>> >
>>
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Caleb Rackliffe]

Kind of in the same place as Benedict/Aleksey.

If we release a 5.1 in, let's say...March of next year, the number of 5.0
users is going to be very minimal. Nobody is going to upgrade anything
important from now through the first half of January anyway, right? They're
going to be making sure their existing clusters aren't exploding.

(We still want TCM/Accord to be available to people to test by Summit, but
that feels unrelated to whether we cut a 5.1 branch...)

Aside: If I had to pick a month of the year to release software used by
large enterprises, it probably would be something like March instead of
December. I have no good research to back that up, of course...

On Mon, Oct 23, 2023 at 12:19 PM Benedict  wrote:

> To be clear, I’m not making an argument either way about the path forwards
> we should take, just concurring about a likely downside of this proposal. I
> don’t have a strong opinion about how we should proceed.
>
> On 23 Oct 2023, at 18:16, Benedict  wrote:
>
> 
> I agree. If we go this route we should essentially announce an immediate
> 5.1 alpha at the same time as 5.0 GA, and I can’t see almost anybody
> rolling out 5.0 with 5.1 so close on its heels.
>
> On 23 Oct 2023, at 18:11, Aleksey Yeshchenko  wrote:
>
> I’m not so sure that many folks will choose to go 4.0->5.0->5.1 path
> instead of just waiting longer for TCM+Accord to be in, and go 4.0->5.1 in
> one hop.
>
> Nobody likes going through these upgrades. So I personally expect 5.0 to
> be a largely ghost release if we go this route, adopted by few, just a
> permanent burden on the merge path to trunk.
>
> Not to say that there isn’t valuable stuff in 5.0 without TCM and Accord -
> there most certainly is - but with the expectation that 5.1 will follow up
> reasonably shortly after with all that *and* two highly anticipated
> features on top, I just don’t see the point. It will be another 2.2 release.
>
>
> On 23 Oct 2023, at 17:43, Josh McKenzie  wrote:
>
> We discussed that at length in various other mailing threads Jeff - kind
> of settled on "we're committing to cutting a major (semver MAJOR or MINOR)
> every 12 months but want to remain flexible for exceptions when
> appropriate".
>
> And then we discussed our timeline for 5.0 this year and settled on the
> "let's try and get it out this calendar year so it's 12 months after 4.1,
> but we'll grandfather in TCM and Accord past freeze date if they can make
> it by October".
>
> So that's the history for how we landed here.
>
> 2) Do we drop the support of 3.0 and 3.11 after 5.0.0 is out or after
> 5.1.0 is?
>
> This is my understanding, yes. Deprecation and support drop is predicated
> on the 5.0 release, not any specific features or anything.
>
> On Mon, Oct 23, 2023, at 12:29 PM, Jeff Jirsa wrote:
>
>
>
> On Mon, Oct 23, 2023 at 4:52 AM Mick Semb Wever  wrote:
>
>
> The TCM work (CEP-21) is in its review stage but being well past our
> cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I would
> like to propose the following.
>
>
>
> I think this presumes that 5.0 GA is date driven instead of feature driven.
>
> I'm sure there's a conversation elsewhere, but why isn't this date movable?
>
>
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Caleb Rackliffe]

...or like the end of January. Either way, feel free to ignore the "aside"
:)

On Mon, Oct 23, 2023 at 12:53 PM Caleb Rackliffe 
wrote:

> Kind of in the same place as Benedict/Aleksey.
>
> If we release a 5.1 in, let's say...March of next year, the number of 5.0
> users is going to be very minimal. Nobody is going to upgrade anything
> important from now through the first half of January anyway, right? They're
> going to be making sure their existing clusters aren't exploding.
>
> (We still want TCM/Accord to be available to people to test by Summit, but
> that feels unrelated to whether we cut a 5.1 branch...)
>
> Aside: If I had to pick a month of the year to release software used by
> large enterprises, it probably would be something like March instead of
> December. I have no good research to back that up, of course...
>
> On Mon, Oct 23, 2023 at 12:19 PM Benedict  wrote:
>
>> To be clear, I’m not making an argument either way about the path
>> forwards we should take, just concurring about a likely downside of this
>> proposal. I don’t have a strong opinion about how we should proceed.
>>
>> On 23 Oct 2023, at 18:16, Benedict  wrote:
>>
>> 
>> I agree. If we go this route we should essentially announce an immediate
>> 5.1 alpha at the same time as 5.0 GA, and I can’t see almost anybody
>> rolling out 5.0 with 5.1 so close on its heels.
>>
>> On 23 Oct 2023, at 18:11, Aleksey Yeshchenko  wrote:
>>
>> I’m not so sure that many folks will choose to go 4.0->5.0->5.1 path
>> instead of just waiting longer for TCM+Accord to be in, and go 4.0->5.1 in
>> one hop.
>>
>> Nobody likes going through these upgrades. So I personally expect 5.0 to
>> be a largely ghost release if we go this route, adopted by few, just a
>> permanent burden on the merge path to trunk.
>>
>> Not to say that there isn’t valuable stuff in 5.0 without TCM and Accord
>> - there most certainly is - but with the expectation that 5.1 will follow
>> up reasonably shortly after with all that *and* two highly anticipated
>> features on top, I just don’t see the point. It will be another 2.2 release.
>>
>>
>> On 23 Oct 2023, at 17:43, Josh McKenzie  wrote:
>>
>> We discussed that at length in various other mailing threads Jeff - kind
>> of settled on "we're committing to cutting a major (semver MAJOR or MINOR)
>> every 12 months but want to remain flexible for exceptions when
>> appropriate".
>>
>> And then we discussed our timeline for 5.0 this year and settled on the
>> "let's try and get it out this calendar year so it's 12 months after 4.1,
>> but we'll grandfather in TCM and Accord past freeze date if they can make
>> it by October".
>>
>> So that's the history for how we landed here.
>>
>> 2) Do we drop the support of 3.0 and 3.11 after 5.0.0 is out or after
>> 5.1.0 is?
>>
>> This is my understanding, yes. Deprecation and support drop is predicated
>> on the 5.0 release, not any specific features or anything.
>>
>> On Mon, Oct 23, 2023, at 12:29 PM, Jeff Jirsa wrote:
>>
>>
>>
>> On Mon, Oct 23, 2023 at 4:52 AM Mick Semb Wever  wrote:
>>
>>
>> The TCM work (CEP-21) is in its review stage but being well past our
>> cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I would
>> like to propose the following.
>>
>>
>>
>> I think this presumes that 5.0 GA is date driven instead of feature
>> driven.
>>
>> I'm sure there's a conversation elsewhere, but why isn't this date
>> movable?
>>
>>
>>

Re: [DISCUSS] Update default disk_access_mode to mmap_index_only on 5.0

[Caleb Rackliffe]

Added one nit to the PR. Otherwise, this is awesome :)

On Wed, Nov 15, 2023 at 11:01 AM Jordan West  wrote:

> I would also like to back this proposal. We change this default because
> several incidents have occurred by leaving the default of auto. There are
> rare cases where auto/mmap is the better option but as for a default
> mmap_index_only is safer.
>
> On Wed, Nov 15, 2023 at 6:35 AM Paulo Motta  wrote:
>
>> Hi,
>>
>> I would like to get back to this. I proposed this default configuration
>> change on the user list ~1 month ago and there were no comments [1].
>>
>> I created CASSANDRA-19021 [2] to make the proposed change and Stefan
>> kindly submitted a patch, CI is looking good.
>>
>> Any objections to making this change in 5.0? If not, we will merge in 24
>> hours.
>>
>> Thanks,
>>
>> Paulo
>>
>> [1] - https://lists.apache.org/thread/w0gkdj7fhylycqwmd73p0kfck7jr8qth
>> [2] - https://issues.apache.org/jira/browse/CASSANDRA-19021
>>
>> On Wed, Sep 6, 2023 at 5:12 PM Paulo Motta 
>> wrote:
>>
>>> > I wonder why disk_access_mode property is not in cassandra.yaml
>>> (looking into trunk right now)
>>>
>>> I think there's a prehistoric reason why it was removed but I can't
>>> remember right now.
>>>
>>> > Do you all think we can add it there with brief explanation what each
>>> option does?
>>>
>>> We could reinclude it as long as we provide a clear recommendation on
>>> when to change from the default since this is an advanced setting which
>>> should be rarely changed. But I still think we should provide a more
>>> stable/foolproof default (mmap_index_only) since the current default (mmap)
>>> is known to cause instability in some scenarios.
>>>
>>> Also there is a technicality with changing the default, if we change the
>>> "auto" behavior from mmap to mmap_index_only this may affect users relying
>>> on the default "mmap" behavior. Not sure the best way to address that, is a
>>> big NEWS note sufficient? Even though users are expected to read NEWS when
>>> upgrading we know well not all users read it.
>>>
>>> > Shall we also share this thread with @user?
>>>
>>> Thanks Ekaterina! If we decide to change the default we can run this
>>> through the user@ list to see what the user community thinks.
>>>
>>> On Wed, Sep 6, 2023 at 4:45 PM Ekaterina Dimitrova <
>>> e.dimitr...@gmail.com> wrote:
>>>
 Thanks for starting this discussion, Paulo!

 Shall we also share this thread with @user?

 On Wed, 6 Sep 2023 at 16:35, C. Scott Andreas 
 wrote:

> Supportive of switching the default to mmap_index_only as well.
>
> I don’t have numbers handy to share, but my experience has been
> significantly lower read latency and I wouldn’t run with auto. I’ve also
> not observed substantial heap pressure after switching - it was strictly 
> an
> improvement.
>
> - Scott
>
> —
> Mobile
>
> On Sep 6, 2023, at 8:50 AM, Paulo Motta 
> wrote:
>
> 
>
> Hi,
>
> I've been bitten by OOMs with disk_access_mode:auto/mmap that were
> fixed by changing to disk_access_mode:mmap_index_only. In a particular
> benchmark I got 5x more read throughput on 3.11.x with disk_access_mode:
> mmap_index_only vs disk_access_mode: auto/mmap.
>
> Changing disk_access_mode to mmap_index_only seems to be a common
> recommendation on forums[1][2][3][4] and slack (find by searching
> disk_access_mode in the #cassandra channel on
> https://the-asf.slack.com/).
>
> It's not clear to me when using the default
> disk_access_mode:auto/mmap is beneficial, perhaps only when the read set
> fits in memory? Mick seems to think on CASSANDRA-15531 [5], that
> mmap_index_only has a higher heap cost and should be only used when
> warranted. However it's not uncommon to see people being bitten with OOMs
> or lower read performance due to the default disk_access_mode, so it makes
> me think it's not the best fool-proof default.
>
> Should we consider changing default "auto" behavior of
> "disk_access_mode" to be "mmap_index_only" instead of "mmap" in 5.0 since
> it's likely safer and perhaps more performant?
>
> Thanks,
>
> Paulo
>
> [1]
> https://stackoverflow.com/questions/72272035/troubleshooting-and-fixing-cassandra-oom-issue
> [2] https://phabricator.wikimedia.org/T137419
> [3] https://stackoverflow.com/a/55975471
> [4]
> https://support.datastax.com/s/article/FAQ-Use-of-disk-access-mode-in-DSE-51-and-earlier
> [5] https://issues.apache.org/jira/browse/CASSANDRA-15531
>
>

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[Caleb Rackliffe]

Just to update this thread, Mike has identified the root cause of 19011 and in the process uncovered 2 additional very closely related issues. (Fantastic job fuzzing there!) Fixes for all three issues are in hand, and he continues to test.After some conversation w/ Mick, Alex, and Mike, I feel like cutting a beta1 now is fine as long as we commit to a beta2 before summit that includes 19011. Putting a beta w/ SAI in this state into users’ hands at summit would be very, very bad.On Nov 27, 2023, at 12:21 PM, Mike Adamson  wrote:> Furthermore, we don't even know if it's still an issue after 19034 was committed. It's a difficult one to reproduce because we don't have access to the harry script that generated the error in the first place. I am investigating it but without the original reproduction it may take some time.On Mon, 27 Nov 2023 at 16:19, Mick Semb Wever  wrote:On Mon, 27 Nov 2023 at 16:28, Brandon Williams  wrote:On Mon, Nov 27, 2023 at 9:25 AM Mick Semb Wever  wrote:
>
> It was agreed to move them to 5.0-rc

Where?
Typo, "it" not "them".I'm only talking about 19011.  The others were already 5.0-rc, or infact forward from 5.0.x.Here:  https://issues.apache.org/jira/browse/CASSANDRA-19011?focusedCommentId=17789202&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17789202Furthermore, we don't even know if it's still an issue after 19034 was committed.  We want to figure this out before the vote window closes. 
-- Mike AdamsonEngineering+1 650 389 6000 | datastax.comFind DataStax Online:        

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[Caleb Rackliffe]

I'm fine w/ alpha2 now and beta1 once we resolve 19011.

On Tue, Nov 28, 2023 at 12:36 PM Benjamin Lerer  wrote:

> -1 based on the problems raised by Caleb.
>
> I would be fine with releasing that version as an alpha as Jeremiah
> proposed.
>
> As of this time, I'm also not aware of a user of the project operating a
>> build from the 5.0 branch at substantial scale to suss out the operational
>> side of what can be expected. If someone is running a build supporting
>> non-perf-test traffic derived from the 5.0 branch and has an experience
>> report to share it would be great to read.
>
>
> Some people at Datastax are working on such testing. It will take a bit of
> time before we get the final results though.
>
> Le mar. 28 nov. 2023 à 19:27, J. D. Jordan  a
> écrit :
>
>> That said. This is clearly better than and with many fixes from the
>> alpha. Would people be more comfortable if this cut was released as another
>> alpha and we do beta1 once the known fixes land?
>>
>> On Nov 28, 2023, at 12:21 PM, J. D. Jordan 
>> wrote:
>>
>> 
>> -0 (NB) on this cut. Given the concerns expressed so far in the thread I
>> would think we should re-cut beta1 at the end of the week.
>>
>> On Nov 28, 2023, at 12:06 PM, Patrick McFadin  wrote:
>>
>> 
>> I'm a +1 on a beta now vs maybe later. Beta doesn't imply perfect
>> especially if there are declared known issues. We need people outside of
>> this tight group using it and finding issues. I know how this rolls. Very
>> few people touch a Alpha release. Beta is when the engine starts and we
>> need to get it started asap. Otherwise we are telling ourselves we have the
>> perfect testing apparatus and don't need more users testing. I don't think
>> that is the case.
>>
>> Scott, Ekaterina, and I are going to be on stage in 2 weeks talking about
>> Cassandra 5 in the keynotes. In that time, our call to action is going to
>> be to test the beta.
>>
>> Patrick
>>
>> On Tue, Nov 28, 2023 at 9:41 AM Mick Semb Wever  wrote:
>>
>>> The vote will be open for 72 hours (longer if needed). Everyone who has
 tested the build is invited to vote. Votes by PMC members are considered
 binding. A vote passes if there are at least three binding +1s and no -1's.

>>>
>>>
>>> +1
>>>
>>> Checked
>>> - signing correct
>>> - checksums are correct
>>> - source artefact builds (JDK 11+17)
>>> - binary artefact runs (JDK 11+17)
>>> - debian package runs (JDK 11+17)
>>> - debian repo runs (JDK 11+17)
>>> - redhat* package runs (JDK11+17)
>>> - redhat* repo runs (JDK 11+17)
>>>
>>>
>>> With the disclaimer:  There's a few known bugs in SAI, e.g. 19011, with
>>> fixes to be available soon in 5.0-beta2.
>>>
>>>
>>>

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[Caleb Rackliffe]

If the consensus is that meaningful community testing will occur in the
week between "beta1 but SAI is broken, friends" and "ok, beta2, it's fixed
now, go for it"...then go for it.

On Tue, Nov 28, 2023 at 12:40 PM Patrick McFadin  wrote:

> JD, that wasn't my point. It feels like we are treating a beta like an RC,
> which it isn't. Ship Beta 1 now and Beta 2 later. We need people looking
> today because they will find new bugs and the signal is lost on alpha. It's
> too yolo for most people.
>
> On Tue, Nov 28, 2023 at 10:36 AM Benjamin Lerer  wrote:
>
>> -1 based on the problems raised by Caleb.
>>
>> I would be fine with releasing that version as an alpha as Jeremiah
>> proposed.
>>
>> As of this time, I'm also not aware of a user of the project operating a
>>> build from the 5.0 branch at substantial scale to suss out the operational
>>> side of what can be expected. If someone is running a build supporting
>>> non-perf-test traffic derived from the 5.0 branch and has an experience
>>> report to share it would be great to read.
>>
>>
>> Some people at Datastax are working on such testing. It will take a bit
>> of time before we get the final results though.
>>
>> Le mar. 28 nov. 2023 à 19:27, J. D. Jordan  a
>> écrit :
>>
>>> That said. This is clearly better than and with many fixes from the
>>> alpha. Would people be more comfortable if this cut was released as another
>>> alpha and we do beta1 once the known fixes land?
>>>
>>> On Nov 28, 2023, at 12:21 PM, J. D. Jordan 
>>> wrote:
>>>
>>> 
>>> -0 (NB) on this cut. Given the concerns expressed so far in the thread I
>>> would think we should re-cut beta1 at the end of the week.
>>>
>>> On Nov 28, 2023, at 12:06 PM, Patrick McFadin 
>>> wrote:
>>>
>>> 
>>> I'm a +1 on a beta now vs maybe later. Beta doesn't imply perfect
>>> especially if there are declared known issues. We need people outside of
>>> this tight group using it and finding issues. I know how this rolls. Very
>>> few people touch a Alpha release. Beta is when the engine starts and we
>>> need to get it started asap. Otherwise we are telling ourselves we have the
>>> perfect testing apparatus and don't need more users testing. I don't think
>>> that is the case.
>>>
>>> Scott, Ekaterina, and I are going to be on stage in 2 weeks talking
>>> about Cassandra 5 in the keynotes. In that time, our call to action is
>>> going to be to test the beta.
>>>
>>> Patrick
>>>
>>> On Tue, Nov 28, 2023 at 9:41 AM Mick Semb Wever  wrote:
>>>
 The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no 
> -1's.
>


 +1

 Checked
 - signing correct
 - checksums are correct
 - source artefact builds (JDK 11+17)
 - binary artefact runs (JDK 11+17)
 - debian package runs (JDK 11+17)
 - debian repo runs (JDK 11+17)
 - redhat* package runs (JDK11+17)
 - redhat* repo runs (JDK 11+17)


 With the disclaimer:  There's a few known bugs in SAI, e.g. 19011, with
 fixes to be available soon in 5.0-beta2.

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[Caleb Rackliffe]

> So in the context of this thread, if I want to try out SAI for example, I
don't care as much about consistency edge cases around coordinators or
replicas or read repair.

That would apply to 19018, not 19011, which is a critical functionality
issue.

On Wed, Nov 29, 2023 at 12:49 PM Jeremy Hanna 
wrote:

> I want to just follow up with functional versus production-worthy.  If I'm
> a user interested in C* 5 and want to try it out as betas come out, I'm
> looking more for something functional and not perfect.  So in the context
> of this thread, if I want to try out SAI for example, I don't care as much
> about consistency edge cases around coordinators or replicas or read
> repair.  I care a lot about that for a RC or GA release but doing POCs with
> betas that have known edge case issues like that is fine IMO.
>
> I know this is likely a moot point for this release since the fixes are
> almost in, but I think just publishing beta 1 and then a follow up beta 2
> with those fixes would be fine in that context, if I understood the bugs
> correctly.
>
> On Nov 29, 2023, at 12:15 PM, Aaron Ploetz  wrote:
>
> Even though my opinion doesn't really count here, I do feel compelled to
> mention that:
>
>  - No one expects a "beta" release to be perfect, but it does signal that
> it is "close" to being ready.
>  - An "alpha" release is in fact a LOT scarier than a "beta" release.
>
> From a user perspective, if I was coaching dev teams on selecting a build
> based on newly available features, I would help them build up a dev/stage
> cluster based on a beta (and make the "beta" part very clear to them).
> However an alpha version just doesn't convey the same level of confidence.
> When I test out an "alpha" of anything, I fully expect some things to just
> be broken.
>
> As for cutting a beta for the Summit; it makes sense that we'd want to get
> some things fixed up before that. But it would also be great to be at the
> point where we have a beta ready for folks to take a look at. We absolutely
> could tell everyone to download the alpha and give it a spin. But more
> people will be likely to do that for a beta than for an alpha.
>
> Take that however you will.
>
> Thanks,
>
> Aaron
>
>
> On Wed, Nov 29, 2023 at 9:54 AM Aleksey Yeshchenko 
> wrote:
>
>> -1 on cutting a beta1 in this state. An alpha2 would be acceptable now,
>> but I’m not sure there is significant value to be had from it. Merge the
>> fixes for outstanding issues listed above, then cut beta1.
>>
>> With TCM and Accord pushed into 5.1, SAI is the headliner user-visible
>> feature. It is what we want users to test. If we are to drive more people
>> to test SAI, we should resolve the issues that we ourselves know of first.
>> Respect our users’ time and effort - don’t make them bump into known bugs.
>>
>> P.S. I don’t believe that ‘alpha' vs. ‘beta' really makes a significant
>> difference to people’s willingness to try out the build. For most folks
>> both feel too raw to play with, and most of the rest would be equally
>> adventurous enough for an alpha *or* a beta. This is just my gut feeling
>> vs. your gut feeling, in absence of hard data.
>>
>> On 28 Nov 2023, at 21:17, Mick Semb Wever  wrote:
>>
>>
>>
>> So then cutting an alpha2 is possible.
>>>
>>
>>
>> Possible, but still leaves alpha1 as our mitigation plan and alpha2 as
>> our best plan.  Doesn't seem worth it IMHO.
>>
>>
>>
>

Re: [VOTE] Release Apache Cassandra 5.0-beta1 (take2)

[Caleb Rackliffe]

+1 (nb)

> On Dec 1, 2023, at 7:32 AM, Mick Semb Wever  wrote:
> 
> 
> 
> Proposing the test build of Cassandra 5.0-beta1 for release.
> 
> sha1: 87fd1fa88a0c859cc32d9f569ad09ad0b345e465
> Git: https://github.com/apache/cassandra/tree/5.0-beta1-tentative
> Maven Artifacts: 
> https://repository.apache.org/content/repositories/orgapachecassandra-1321/org/apache/cassandra/cassandra-all/5.0-beta1/
> 
> The Source and Build Artifacts, and the Debian and RPM packages and 
> repositories, are available here: 
> https://dist.apache.org/repos/dist/dev/cassandra/5.0-beta1/
> 
> The vote will be open for 72 hours (longer if needed). Everyone who has 
> tested the build is invited to vote. Votes by PMC members are considered 
> binding. A vote passes if there are at least three binding +1s and no -1's.
> 
> [1]: CHANGES.txt: 
> https://github.com/apache/cassandra/blob/5.0-beta1-tentative/CHANGES.txt
> [2]: NEWS.txt: 
> https://github.com/apache/cassandra/blob/5.0-beta1-tentative/NEWS.txt

Re: [DISCUSS] CASSANDRA-18940 SAI post-filtering reads don't update local table latency metrics

[Caleb Rackliffe]

Option 1 would be my preference. Seems both useful to have a single metric
for read load against the table and a way to break out SAI reads
specifically.

On Fri, Dec 1, 2023 at 11:00 AM Mike Adamson  wrote:

> Hi,
>
> We are looking at adding SAI post-filtering reads to the local table
> metrics and would like some feedback on the best approach.
>
> We don't think that SAI reads are that special so they can be included in
> the table latencies, but how do we handle the global counts and the SAI
> counts? Do we need to maintain a separate count of SAI reads? We feel the
> answer to this is yes so how do we do the counting? There are two options
> (others welcome):
>
> 1. All reads go into the current global count and we have a separate count
> for SAI specific reads. So non-SAI reads = global count - SAI count
> 2. We leave the exclude the SAI reads from the current global count so
> total reads = global count + SAI count
>
> Our preference is for option 1 above. Does anyone have any strong views /
> opinions on this?
>
>
>
> --
> [image: DataStax Logo Square]  *Mike Adamson*
> Engineering
>
> +1 650 389 6000 <16503896000> | datastax.com 
> Find DataStax Online: [image: LinkedIn Logo]
> 
>[image: Facebook Logo]
> 
>[image: Twitter Logo]    [image: RSS
> Feed]    [image: Github Logo]
> 
>
>

Re: [DISCUSS] CASSANDRA-18940 SAI post-filtering reads don't update local table latency metrics

[Caleb Rackliffe]

So the plan would be to have local "Read" and "Range" remain unchanged in
TableMetrics, but have a third "SAIRead" (?) just for SAI post-filtering
read SinglePartitionReadCommands? I won't complain too much if that's what
we settle on, but it just depends on how much this is a metric for
ReadCommand subclasses operating at the node-local level versus something
we think we should link conceptually to a user query. SAI queries will
produce a SinglePartitionReadCommand per matching primary key, so that
definitely won't work for the latter.

@Mike On a related note, we now have "PartitionReads" and "RowsFiltered" in
TableQueryMetrics. Should the former just be removed, given a.) it actually
is rows now not partitions and b.) "RowsFiltered" seems like it'll be
almost  the same thing now? (I guess if we ever try batching rows reads per
partition, it would come in handy again...)

On Fri, Dec 1, 2023 at 12:30 PM J. D. Jordan 
wrote:

> I prefer option 2. It is much easier to understand and roll up two metrics
> than to do subtractive dashboards.
>
> SAI reads are already “range reads” for the client level metrics, not
> regular reads. So grouping them into the regular read metrics at the lower
> level seems confusing to me in that sense as well.
>
> As an operator I want to know how my SAI reads and normal reads are
> performing latency wise separately.
>
> -Jeremiah
>
> On Dec 1, 2023, at 11:15 AM, Caleb Rackliffe 
> wrote:
>
> 
> Option 1 would be my preference. Seems both useful to have a single metric
> for read load against the table and a way to break out SAI reads
> specifically.
>
> On Fri, Dec 1, 2023 at 11:00 AM Mike Adamson 
> wrote:
>
>> Hi,
>>
>> We are looking at adding SAI post-filtering reads to the local table
>> metrics and would like some feedback on the best approach.
>>
>> We don't think that SAI reads are that special so they can be included in
>> the table latencies, but how do we handle the global counts and the SAI
>> counts? Do we need to maintain a separate count of SAI reads? We feel the
>> answer to this is yes so how do we do the counting? There are two options
>> (others welcome):
>>
>> 1. All reads go into the current global count and we have a separate
>> count for SAI specific reads. So non-SAI reads = global count - SAI count
>> 2. We leave the exclude the SAI reads from the current global count so
>> total reads = global count + SAI count
>>
>> Our preference is for option 1 above. Does anyone have any strong views /
>> opinions on this?
>>
>>
>>
>> --
>> [image: DataStax Logo Square] <https://www.datastax.com/> *Mike Adamson*
>> Engineering
>>
>> +1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
>> Find DataStax Online: [image: LinkedIn Logo]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
>>[image: Facebook Logo]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
>>[image: Twitter Logo] <https://twitter.com/DataStax>   [image: RSS
>> Feed] <https://www.datastax.com/blog/rss.xml>   [image: Github Logo]
>> <https://github.com/datastax>
>>
>>

Re: [DISCUSS] CASSANDRA-18940 SAI post-filtering reads don't update local table latency metrics

[Caleb Rackliffe]

Right. SAI queries are distributed range queries that produce local
single-partition reads. They should absolutely not be recorded in the local
range read latency metric. I'm fine ultimately with a new metric or the
existing local single-partition read metric.

On Fri, Dec 1, 2023 at 1:02 PM J. D. Jordan 
wrote:

> At the coordinator level SAI queries fall under Range metrics. I would
> either put them under the same at the lower level or in a new SAI metric.
>
> It would be confusing to have the top level coordinator query metrics in
> Range and the lower level in Read.
>
> On Dec 1, 2023, at 12:50 PM, Caleb Rackliffe 
> wrote:
>
> 
> So the plan would be to have local "Read" and "Range" remain unchanged in
> TableMetrics, but have a third "SAIRead" (?) just for SAI post-filtering
> read SinglePartitionReadCommands? I won't complain too much if that's what
> we settle on, but it just depends on how much this is a metric for
> ReadCommand subclasses operating at the node-local level versus something
> we think we should link conceptually to a user query. SAI queries will
> produce a SinglePartitionReadCommand per matching primary key, so that
> definitely won't work for the latter.
>
> @Mike On a related note, we now have "PartitionReads" and "RowsFiltered"
> in TableQueryMetrics. Should the former just be removed, given a.) it
> actually is rows now not partitions and b.) "RowsFiltered" seems like it'll
> be almost  the same thing now? (I guess if we ever try batching rows reads
> per partition, it would come in handy again...)
>
> On Fri, Dec 1, 2023 at 12:30 PM J. D. Jordan 
> wrote:
>
>> I prefer option 2. It is much easier to understand and roll up two
>> metrics than to do subtractive dashboards.
>>
>> SAI reads are already “range reads” for the client level metrics, not
>> regular reads. So grouping them into the regular read metrics at the lower
>> level seems confusing to me in that sense as well.
>>
>> As an operator I want to know how my SAI reads and normal reads are
>> performing latency wise separately.
>>
>> -Jeremiah
>>
>> On Dec 1, 2023, at 11:15 AM, Caleb Rackliffe 
>> wrote:
>>
>> 
>> Option 1 would be my preference. Seems both useful to have a single
>> metric for read load against the table and a way to break out SAI reads
>> specifically.
>>
>> On Fri, Dec 1, 2023 at 11:00 AM Mike Adamson 
>> wrote:
>>
>>> Hi,
>>>
>>> We are looking at adding SAI post-filtering reads to the local table
>>> metrics and would like some feedback on the best approach.
>>>
>>> We don't think that SAI reads are that special so they can be included
>>> in the table latencies, but how do we handle the global counts and the SAI
>>> counts? Do we need to maintain a separate count of SAI reads? We feel the
>>> answer to this is yes so how do we do the counting? There are two options
>>> (others welcome):
>>>
>>> 1. All reads go into the current global count and we have a separate
>>> count for SAI specific reads. So non-SAI reads = global count - SAI count
>>> 2. We leave the exclude the SAI reads from the current global count so
>>> total reads = global count + SAI count
>>>
>>> Our preference is for option 1 above. Does anyone have any strong views
>>> / opinions on this?
>>>
>>>
>>>
>>> --
>>> [image: DataStax Logo Square] <https://www.datastax.com/> *Mike Adamson*
>>> Engineering
>>>
>>> +1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
>>> Find DataStax Online: [image: LinkedIn Logo]
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
>>>[image: Facebook Logo]
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
>>>[image: Twitter Logo] <https://twitter.com/DataStax>   [image: RSS
>>> Feed] <https://www.datastax.com/blog/rss.xml>   [image: Github Logo]
>>> <https://github.com/datastax>
>>>
>>>

Re: Welcome Mike Adamson as Cassandra committer

[Caleb Rackliffe]

Congratulations! Very well deserved.

On Fri, Dec 8, 2023 at 10:33 AM Mick Semb Wever  wrote:

> Congrats Mike !!
>
>>

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Caleb Rackliffe]

+1

Agree w/ all the justifications mentioned above.

As a reviewer on CASSANDRA-19210
, my goals were to
a.) look at the directory, naming, and package structure of the ported
code, b.) make sure IDE integration was working, and c.) make sure any
modifications to existing code (rather than direct code movements from
cassandra-harry) were straightforward.

On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov  wrote:

> Hey folks,
>
> I am mostly done with a patch that brings Harry in-tree [1]. I will
> trigger one more CI run overnight, and my intention was to merge it some
> time soon, but I wanted to give a fair warning here, since this is a
> relatively large patch.
>
> Good news for everyone that it:
>   a) touches no production code whatsoever. Only test (in-jvm dtest
> namely) code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate
> version of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to
> have some base for examples/usage.
>
> Since we were effectively relying on this code for a while now, and the
> intention now is to converge to:
>   a) fewer different generators, and have a shareable version of
> generators for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex
> scenarios
> myself and many other Cassandra contributors have expressed an opinion
> that bringing Harry in-tree will be highly benefitial.
>
> I strongly believe that bringing Harry in-tree will help to lower the
> barrier for fuzz test and simplify co-development of Cassandra and Harry.
> Previously, it has been rather difficult to debug edge cases because I had
> to either re-compile an in-jvm dtest jar and bring it to Harry, or
> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
> time consuming. Moreover, I believe we have missed at very least one RT
> regression [2] because Harry was not in-tree, as its tests would've caught
> the issue even with the model that existed.
>
> For other recently found issues, I think having Harry in-tree would have
> substantially lowered a turnaround time, and allowed me to share repros
> with developers of corresponding features much quicker.
>
> I do expect a slight learning curve for Harry, but my intention is to
> build a web of simple tests (worked on some of them yesterday after
> conversation with David already), which can follow the in-jvm-dtest pattern
> of find-similar-test / copy / modify. There's already copious
> documentation, so I do not believe not having docs for Harry was ever an
> issue, since there have been plenty.
>
> You all are aware of my dedication to testing and quality of Apache
> Cassandra, and I hope you also see the benefits of having a model checker
> in-tree.
>
> Thank you and happy upcoming holidays,
> --Alex
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>
>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Caleb Rackliffe]

> We are also currently working on some SAI features that need cost based
optimization.

I don't even think we have to think about *new* SAI features to see where
it will benefit from further *local* optimization, and I'm sympathetic to
that happening in the context of a larger framework, as long as the
framework itself starts as thin as possible and grows over time.

For SAI, the main difficulties we're likely to have in the very short term
are a.) how to order/choose predicates during AND queries to minimize
intersection complexity, b.) how to make decisions about when to use an
index or simple filtering, and c.) combinations of those two, where we
might take different paths depending on how many predicates exist and the
cardinality of the term indexes those predicates touch.

ex. We have a system property called SAI_INTERSECTION_CLAUSE_LIMIT (in CRP)
that controls the maximum number of index query intersections that will
participate in an AND query, leaving the rest for post-filtering. Having
local cardinality estimation on the individual column indexes might make it
a lot easier to pick the two most selective predicates. (Numeric range
predicates, for example, can have matching posting lists of wildly varying
sizes.)

tl;dr I'd like to see us start by enumerating the specific scenarios where
query optimization will benefit SAI in conjunction w/ creating a template
for how a high-level CBO apparatus would work (which is sort of what we
have in this CEP, even if it doesn't go into extreme detail). Then, we
build from the bottom up to ship improvements as quickly as possible w/o
compromising the longer term CBO vision.

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Caleb Rackliffe]

What would the relationship between our present query tracing apparatus and
EXPLAIN ANALYZE look like?

On Thu, Dec 21, 2023 at 4:24 PM Caleb Rackliffe 
wrote:

> > We are also currently working on some SAI features that need cost based
> optimization.
>
> I don't even think we have to think about *new* SAI features to see where
> it will benefit from further *local* optimization, and I'm sympathetic to
> that happening in the context of a larger framework, as long as the
> framework itself starts as thin as possible and grows over time.
>
> For SAI, the main difficulties we're likely to have in the very short term
> are a.) how to order/choose predicates during AND queries to minimize
> intersection complexity, b.) how to make decisions about when to use an
> index or simple filtering, and c.) combinations of those two, where we
> might take different paths depending on how many predicates exist and the
> cardinality of the term indexes those predicates touch.
>
> ex. We have a system property called SAI_INTERSECTION_CLAUSE_LIMIT (in
> CRP) that controls the maximum number of index query intersections that
> will participate in an AND query, leaving the rest for post-filtering.
> Having local cardinality estimation on the individual column indexes might
> make it a lot easier to pick the two most selective predicates. (Numeric
> range predicates, for example, can have matching posting lists of wildly
> varying sizes.)
>
> tl;dr I'd like to see us start by enumerating the specific scenarios where
> query optimization will benefit SAI in conjunction w/ creating a template
> for how a high-level CBO apparatus would work (which is sort of what we
> have in this CEP, even if it doesn't go into extreme detail). Then, we
> build from the bottom up to ship improvements as quickly as possible w/o
> compromising the longer term CBO vision.
>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Caleb Rackliffe]

I think I hinted at this in my first response, but just to clarify, I would
be interested to see this work broken up as much as possible into a.) the
set of things we can do without coordinator involvement (statistical
optimization for index and filtering queries) and b.) the set of things
where coordinator-level planning is harder to avoid (joins?)

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe]

Hey all,

I'm a bit late to the discussion. I see that we've already discussed
CASSANDRA-15013  and
CASSANDRA-16663  at
least in passing. Having written the latter, I'd be the first to admit it's
a crude tool, although it's been useful here and there, and provides a
couple primitives that may be useful for future work. As Scott mentions,
while it is configurable at runtime, it is not adaptive, although we did
make configuration easier in CASSANDRA-17423
. It also is global
to the node, although we've lightly discussed some ideas around making it
more granular. (For example, keyspace-based limiting, or limiting "domains"
tagged by the client in requests, could be interesting.) It also does not
deal with inter-node traffic, of course.

Something we've not yet mentioned (that does address internode traffic) is
CASSANDRA-17324 ,
which I proposed shortly after working on the native request limiter (and
have just not had much time to return to). The basic idea is this:

When a node is struggling under the weight of a compaction backlog and
> becomes a cause of increased read latency for clients, we have two safety
> valves:
>
> 1.) Disabling the native protocol server, which stops the node from
> coordinating reads and writes.
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>
> These are useful, but we don’t appear to have any mechanism that would
> allow us to temporarily reject internode hint, batch, and mutation messages
> that could further delay resolution of the compaction backlog.
>

Whether it's done as part of a larger framework or on its own, it still
feels like a good idea.

Thinking in terms of opportunity costs here (i.e. where we spend our finite
engineering time to holistically improve the experience of operating this
database) is healthy, but we probably haven't reached the point of
diminishing returns on nodes being able to protect themselves from clients
and from other nodes. I would just keep in mind two things:

1.) The effectiveness of rate-limiting in the system (which includes the
database and all clients) as a whole necessarily decreases as we move from
the application to the lowest-level database internals. Limiting correctly
at the client will save more resources than limiting at the native protocol
server, and limiting correctly at the native protocol server will save more
resources than limiting after we've dispatched requests to some thread pool
for processing.
2.) We should make sure the links between the "known" root causes of
cascading failures and the mechanisms we introduce to avoid them remain
very strong.

In any case, I'd be happy to help out in any way I can as this moves
forward (especially as it relates to our past/current attempts to address
this problem space).

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe]

I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe  wrote:Hey all,I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 and CASSANDRA-16663 at least in passing. Having written the latter, I'd be the first to admit it's a crude tool, although it's been useful here and there, and provides a couple primitives that may be useful for future work. As Scott mentions, while it is configurable at runtime, it is not adaptive, although we did make configuration easier in CASSANDRA-17423. It also is global to the node, although we've lightly discussed some ideas around making it more granular. (For example, keyspace-based limiting, or limiting "domains" tagged by the client in requests, could be interesting.) It also does not deal with inter-node traffic, of course.Something we've not yet mentioned (that does address internode traffic) is CASSANDRA-17324, which I proposed shortly after working on the native request limiter (and have just not had much time to return to). The basic idea is this:When a node is struggling under the weight of a compaction backlog and becomes a cause of increased read latency for clients, we have two safety valves:1.) Disabling the native protocol server, which stops the node from coordinating reads and writes.2.) Jacking up the severity on the node, which tells the dynamic snitch to avoid the node for reads from other coordinators.These are useful, but we don’t appear to have any mechanism that would allow us to temporarily reject internode hint, batch, and mutation messages that could further delay resolution of the compaction backlog.Whether it's done as part of a larger framework or on its own, it still feels like a good idea.Thinking in terms of opportunity costs here (i.e. where we spend our finite engineering time to holistically improve the experience of operating this database) is healthy, but we probably haven't reached the point of diminishing returns on nodes being able to protect themselves from clients and from other nodes. I would just keep in mind two things:1.) The effectiveness of rate-limiting in the system (which includes the database and all clients) as a whole necessarily decreases as we move from the application to the lowest-level database internals. Limiting correctly at the client will save more resources than limiting at the native protocol server, and limiting correctly at the native protocol server will save more resources than limiting after we've dispatched requests to some thread pool for processing.2.) We should make sure the links between the "known" root causes of cascading failures and the mechanisms we introduce to avoid them remain very strong.In any case, I'd be happy to help out in any way I can as this moves forward (especially as it relates to our past/current attempts to address this problem space).

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe]

I just remembered the other day that I had done a quick writeup on the
state of compaction stress-related throttling in the project:

https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing

I'm sure most of it is old news to the people on this thread, but I figured
I'd post it just in case :)

On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie  wrote:

> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> Seems to me that our historical strategy was to address individual known
> cases one-by-one rather than looking for a more holistic load-balancing and
> load-shedding solution. While the engineer in me likes the elegance of a
> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
> wonders how far we think we are today from a stable set-point.
>
> i.e. are we facing a handful of cases where nodes can still get pushed
> over and then cascade that we can surgically address, or are we facing a
> broader lack of back-pressure that rears its head in different domains
> (client -> coordinator, coordinator -> replica, internode with other
> operations, etc) at surprising times and should be considered more
> holistically?
>
> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>
> I almost forgot CASSANDRA-15817, which introduced
> reject_repair_compaction_threshold, which provides a mechanism to stop
> repairs while compaction is underwater.
>
> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe 
> wrote:
>
> 
> Hey all,
>
> I'm a bit late to the discussion. I see that we've already discussed
> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>  and CASSANDRA-16663
> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
> passing. Having written the latter, I'd be the first to admit it's a crude
> tool, although it's been useful here and there, and provides a couple
> primitives that may be useful for future work. As Scott mentions, while it
> is configurable at runtime, it is not adaptive, although we did
> make configuration easier in CASSANDRA-17423
> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
> global to the node, although we've lightly discussed some ideas around
> making it more granular. (For example, keyspace-based limiting, or limiting
> "domains" tagged by the client in requests, could be interesting.) It also
> does not deal with inter-node traffic, of course.
>
> Something we've not yet mentioned (that does address internode traffic) is
> CASSANDRA-17324 <https://issues.apache.org/jira/browse/CASSANDRA-17324>,
> which I proposed shortly after working on the native request limiter (and
> have just not had much time to return to). The basic idea is this:
>
> When a node is struggling under the weight of a compaction backlog and
> becomes a cause of increased read latency for clients, we have two safety
> valves:
>
> 1.) Disabling the native protocol server, which stops the node from
> coordinating reads and writes.
> 2.) Jacking up the severity on the node, which tells the dynamic snitch to
> avoid the node for reads from other coordinators.
>
> These are useful, but we don’t appear to have any mechanism that would
> allow us to temporarily reject internode hint, batch, and mutation messages
> that could further delay resolution of the compaction backlog.
>
>
> Whether it's done as part of a larger framework or on its own, it still
> feels like a good idea.
>
> Thinking in terms of opportunity costs here (i.e. where we spend our
> finite engineering time to holistically improve the experience of operating
> this database) is healthy, but we probably haven't reached the point of
> diminishing returns on nodes being able to protect themselves from clients
> and from other nodes. I would just keep in mind two things:
>
> 1.) The effectiveness of rate-limiting in the system (which includes the
> database and all clients) as a whole necessarily decreases as we move from
> the application to the lowest-level database internals. Limiting correctly
> at the client will save more resources than limiting at the native protocol
> server, and limiting correctly at the native protocol server will save more
> resources than limiting after we've dispatched requests to some thread pool
> for processing.
> 2.) We should make sure the links between the "known" root causes of
> cascading failures and the mechanisms we introduce to avoid them remain
> very strong.
>
> In any case, I'd be happy to help out in any way I can as this moves
> forward (especially as it relates to our past/current attempts to address
> this problem space).
>
>
>

Re: [DISCUSS] What SHOULD we do when we index an inet type that is ipv4?

[Caleb Rackliffe]

Yeah, what we have with inet is much like if we had a type like "numeric"
that allowed you to write both ints and doubles. If we had actual "inet4"
and "inet6" types, SAI would have been able to index them as fixed length
values without doing the 4 -> 16 byte conversion. Given SAI could easily
change this to go one way or another at post-filtering time, perhaps
there's another option:

4.) Have an option on the column index that allows the user to specify
whether ipv4 and ipv6 addresses are comparable. If they are, nothing
changes. If they aren't, we can just take the matches from the index and
filter "strictly".

I'm not sure what's best here, because what it seems to hinge on is what
users actually want to do when they throw both v4 and v6 addresses into a
single column. Without any real loss in storage efficiency, you could index
them in two separate columns on the same table, and none of this matters.
If they are mixed, it feels like we should at least have the option to make
them comparable, kind of like we have the option to make text
case-insensitive or unicode normalized right now.

On Wed, Mar 6, 2024 at 4:35 PM Bowen Song via dev 
wrote:

> Technically, 127.0.0.1 (IPv4) is not 0:0:0:0:0::7f00:0001 (IPv6),
> but their values are equal. Just like 1.0 (double) is not 1 (int), but
> their values are equal. So, what is the meaning of "=" in CQL?
>
> On 06/03/2024 21:36, David Capwell wrote:
> > So, was reviewing SAI and found we convert ipv4 to ipv6 (which is valid
> for the type) and made me wonder what the behavior would be if client mixed
> ipv4 with ipv4 encoded as ipv6… this caused me to find a different behavior
> in SAI to the rest of C*… where I feel C* is doing the wrong thing…
> >
> > Lets walk over a simple example
> >
> > ipv4: 127.0.0.1
> > ipv6: 0:0:0:0:0::7f00:0001
> >
> > Both of these address are equal according to networking and java… but
> for C* they are different!  These are 2 different values as ipv4 is 4 bytes
> and ipv6 is 16 bytes, so 4 != 16!
> >
> > With SAI we convert all ipv4 to ipv6 so that the search logic is
> correct… this causes SAI to return partitions that ALLOW FILTERING and
> other indexes wouldn’t…
> >
> > This gets to the question in the subject… what SHOULD we do for this
> type?
> >
> > I see 3 options:
> >
> > 1) SAI use the custom C* semantics where 4 != 16… this keeps us
> consistent…
> > 2) ALLOW FILTERING and other indexes are “fixed” so that we actually
> match correctly… we are not really able to fix if the type is in a
> partition or clustering column though…
> > 3) deprecate inet in favor of a inet_better type… where inet semantics
> is the custom C* semantics and inet_better handles this case
> >
> > Thoughts?
>

Re: [DISCUSS] What SHOULD we do when we index an inet type that is ipv4?

[Caleb Rackliffe]

> if an inet type column is a partition key, can I write to it in IPv4 and
then query it with IPv6 and find the record?

You can't...however...

Especially when the original/existing behavior here was possibly not all
that well-conceived, I think it would at least be a good idea to maintain
an *ability* to query inet columns that is v4/v6 format agnostic. We could
change nothing in the SAI index itself and control this behavior at
post-filtering w/ a few lines of code. (i.e. The default could still be to
compare the raw bytes like we would do w/ a key element.)

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[Caleb Rackliffe]

The vector issues itself was a simple error message change:
https://github.com/datastax/python-driver/commit/e90c0f5d71f4cac94ed80ed72c8789c0818e11d0

Was there something else in 3.29.0 that actually necessitated the move to a
floor of Python 3.8? Do we generally change runtime requirements in minor
releases for the driver?

On Mon, Mar 11, 2024 at 12:12 PM Brandon Williams  wrote:

> Given that 3.6 has been EOL for 2+ years[1], I don't think it makes
> sense to add support for it back.
>
> Kind Regards,
> Brandon
>
> [1] https://devguide.python.org/versions/
>
> On Mon, Mar 11, 2024 at 12:08 PM David Capwell  wrote:
> >
> > Originally we had planned to support RHEL 7 but in testing 5.0 we found
> out that cqlsh no longer works on RHEL 7[1].  This was changed in
> CASSANDRA-19245 which upgraded python-driver from 3.28.0 to 3.29.0. For
> some reason this minor version upgrade also dropped support for python 3.6
> which is the supported python version on RHEL 7.
> >
> > We wanted to bring this to the attention of the community to figure out
> next steps; do we wish to say that RHEL 7 is no longer supported (making
> upgrades tied to OS upgrades, which can be very hard for users), or do we
> want to add python 3.6 support back to python-driver?
> >
> >
> > 1: the error seen by users is
> > $ cqlsh
> > Warning: unsupported version of Python, required 3.8-3.11 but found 3.6
> Warning: unsupported version of Python, required 3.8-3.11 but found 2.7
> > No appropriate Python interpreter found.
> > $
> >
> >
>

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[Caleb Rackliffe]

I can try this out on trunk. Will report back...

On Mon, Mar 11, 2024 at 2:23 PM J. D. Jordan 
wrote:

> The Python driver dropped official support for older EOL Python versions
> because they are EOL and no longer tested by the newer driver CI. I don’t
> think there are actually any changes yet that it won’t work in 3.6 still?
> Maybe someone with Python 3.6 installed can change the if and see?  I think
> we have some cqlsh tests in dtest?  As long as we as a project run those on
> RHEL 7, I would be comfortable with adding that back to being supported.
> Though maybe just in the rpm package?
>
> -Jeremiah
>
> On Mar 11, 2024, at 1:33 PM, Josh McKenzie  wrote:
>
> 
> Looks like we bumped from 3.6 requirement to 3.7 in CASSANDRA-18960
> <https://issues.apache.org/jira/browse/CASSANDRA-18960> as well - similar
> thing. Vector support in python, though that patch took it from "return a
> simple blob" to "return something the python driver knows about, but
> apparently not variable types so we'll need to upgrade again."
>
> The version of the Python driver that is used by cqlsh (3.25.0) doesn't
> entirely support the new vector data type introduced by CASSANDRA-18504
> <https://issues.apache.org/jira/browse/CASSANDRA-18504>. While we can
> perfectly write data, read vectors are presented as blobs:
>
>
> As far as I can tell, support for vector types in cqlsh is the sole reason
> we've bumped to 3.7 and 3.8 to support that python driver. That correct
> Andres / Brandon?
>
> On Mon, Mar 11, 2024, at 1:22 PM, Caleb Rackliffe wrote:
>
> The vector issues itself was a simple error message change:
> https://github.com/datastax/python-driver/commit/e90c0f5d71f4cac94ed80ed72c8789c0818e11d0
>
> Was there something else in 3.29.0 that actually necessitated the move to
> a floor of Python 3.8? Do we generally change runtime requirements in minor
> releases for the driver?
>
> On Mon, Mar 11, 2024 at 12:12 PM Brandon Williams 
> wrote:
>
> Given that 3.6 has been EOL for 2+ years[1], I don't think it makes
> sense to add support for it back.
>
> Kind Regards,
> Brandon
>
> [1] https://devguide.python.org/versions/
>
> On Mon, Mar 11, 2024 at 12:08 PM David Capwell  wrote:
> >
> > Originally we had planned to support RHEL 7 but in testing 5.0 we found
> out that cqlsh no longer works on RHEL 7[1].  This was changed in
> CASSANDRA-19245 which upgraded python-driver from 3.28.0 to 3.29.0. For
> some reason this minor version upgrade also dropped support for python 3.6
> which is the supported python version on RHEL 7.
> >
> > We wanted to bring this to the attention of the community to figure out
> next steps; do we wish to say that RHEL 7 is no longer supported (making
> upgrades tied to OS upgrades, which can be very hard for users), or do we
> want to add python 3.6 support back to python-driver?
> >
> >
> > 1: the error seen by users is
> > $ cqlsh
> > Warning: unsupported version of Python, required 3.8-3.11 but found 3.6
> Warning: unsupported version of Python, required 3.8-3.11 but found 2.7
> > No appropriate Python interpreter found.
> > $
> >
> >
>
>
>

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[Caleb Rackliffe]

I did a quick experiment to revert all the bits that require 3.8+ in the
server codebase (while leaving 3.29.0 in place), and I don't see anything
breaking in the tests on trunk.

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[Caleb Rackliffe]

Alright, so there has been a little conversation in ASF Slack here:
https://the-asf.slack.com/archives/CK23JSY2K/p1710255088441369

It looks like we've coalesced around allowing Python 3.6 and 3.7, as
there's no reason they shouldn't work w/ the 3.29.0 Python driver. However,
we'll warn anyone who does run cqlsh w/ those versions that they are EOL,
support may be removed in a future C* release, and they may be used on an
"as is" basis. I'll get a Jira up shortly...

On Mon, Mar 11, 2024 at 8:51 PM Caleb Rackliffe 
wrote:

> I did a quick experiment to revert all the bits that require 3.8+ in the
> server codebase (while leaving 3.29.0 in place), and I don't see anything
> breaking in the tests on trunk.
>

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[Caleb Rackliffe]

Just created CASSANDRA-19467
<https://issues.apache.org/jira/browse/CASSANDRA-19467>

On Tue, Mar 12, 2024 at 1:45 PM Caleb Rackliffe 
wrote:

> Alright, so there has been a little conversation in ASF Slack here:
> https://the-asf.slack.com/archives/CK23JSY2K/p1710255088441369
>
> It looks like we've coalesced around allowing Python 3.6 and 3.7, as
> there's no reason they shouldn't work w/ the 3.29.0 Python driver. However,
> we'll warn anyone who does run cqlsh w/ those versions that they are EOL,
> support may be removed in a future C* release, and they may be used on an
> "as is" basis. I'll get a Jira up shortly...
>
> On Mon, Mar 11, 2024 at 8:51 PM Caleb Rackliffe 
> wrote:
>
>> I did a quick experiment to revert all the bits that require 3.8+ in the
>> server codebase (while leaving 3.29.0 in place), and I don't see anything
>> breaking in the tests on trunk.
>>
>

Re: [DISCUSSION] Replace the Config class instance with the tree-based framework

[Caleb Rackliffe]

>  I kinda feel this should be separated as we can and do do this today but
the reason we have not grouped is not due to the framework we use but more
“what makes sense to group”

I think that's more or less correct. The critical path thing here is
getting to consensus (in CASSANDRA-17292
) on what we
actually want a future nested YAML that a user will interact with to look
like. How we implement it and how we ensure backwards compatibility/version
the YAML are separate questions (questions that will probably be answered
in part by things like the efficiency of the framework around read costs,
etc.)

On Thu, Mar 14, 2024 at 11:44 AM David Capwell  wrote:

> Just went over the doc and I don’t see a real argument for why we want to
> switch “frameworks”… I think this should be fleshed out more; what is it we
> are actually solving for?
>
> Also, I “think” its trying to propose we switch from a mostly flat yaml to
> a nested structure… I kinda feel this should be separated as we can and do
> do this today but the reason we have not grouped is not due to the
> framework we use but more “what makes sense to group” (lack of agreement
> and consistency here… ).  There was a JIRA that Caleb worked on and him and
> Benedict came up with a proposal, I recommend taking a look at that past
> work if you wish to do something similar.
>
> We all have to maintain our configs, so we need to be able to answer a few
> questions with this type of proposal
>
> 1) how will developers maintain, and does this add any burdens to them?
> What benefits do I get as a maintainer?
> 2) what are the read costs.  During a single CQL command we do many
> accesses to Config to see what we should and should not do/allow, so the
> costs here must be known or easy to understand.
> 3) config is a critical and core part of the code, so a rewrite comes with
> huge risk and cost, do we have equal or greater reward for this?  What are
> those rewards?  What are those risks?
> 4) can the motivations for this work be solved in a different way that
> improves one or more of the above questions?  Why is your proposal more
> desirable than those solutions?
>
> > On Mar 13, 2024, at 12:15 PM, Maxim Muzafarov  wrote:
> >
> > Hello everyone,
> >
> > During the implementation, many valid design comments were made about
> > making the virtual table SettingTable [1] updatable. So, I've
> > rethought the whole concept once again, and I want to take another
> > step forward to make this problem feasible with less effort on our
> > part.
> >
> > I want to replace the way we use and store node configuration values
> > internally, which means I want to replace the Config class instance,
> > where we store values with a tree-based framework.
> >>> I propose to use the Lightbend API to do this. <<
> >
> > The changes themselves are quite limited, they don't require rewriting
> > the whole codebase. All the DatabaseDescriptor methods will be
> > retained, and the only thing that would change is the way we store the
> > values (in the in-memory tree, not in the Config class instance
> > itself). So I don't expect that it will be a huge change.
> >
> >
> > All the design details are in the document below, including the
> > framework comparison, the API, and the way how we will manage the
> > configuration schema.
> >
> > Please take a look, I want to move things forward as every important
> > change pulls on a bigger problem that has been neglected for years :-)
> > Let's agree on the framework/API we want to use so that I can invest
> > more time in the implementation.
> >
> >
> https://docs.google.com/document/d/11W1Qj-6d9ZqHv86iEKgFutcxY2DTMIofEbr-zQiw930/edit#heading=h.2051pbez4rce
> >
> > Looking forward to your comments.
> >
> > [1] https://issues.apache.org/jira/browse/CASSANDRA-15254
>
>

Re: [DISCUSS] NULL handling and the unfrozen collection issue

[Caleb Rackliffe]

The easiest way to check out how Accord uses IS NULL and IS NOT NULL is to
look at the examples in the cep-15-accord branch:

https://github.com/apache/cassandra/blob/cep-15-accord/test/distributed/org/apache/cassandra/distributed/test/accord/AccordCQLTestBase.java

tl;dr We did indeed try to go with an approach that more closely matches
SQL, although there may still be some edges we didn't test.

I'd have no problem w/ moving to 3-value logic everywhere, I guess, but
"everywhere" could just mean filtering queries and Accord. (i.e. If we want
to deprecate LWT col = NULL syntax, do we really want people rewriting
those LWTs...or just moving to the new Accord syntax, which obviously
supports it? We should "spend" our user query rewrite budget wisely.)

On Thu, Apr 4, 2024 at 4:53 AM Benjamin Lerer  wrote:

> Now, in Cassandra setting a column to null means deleting it and if *all*
>> columns in a row are null the row is deleted. This might be another edge
>> case...
>
>
> It is slightly more complicated than that as the primary key columns count
> in (*all* columns)
>
> For example if you have the following table: CREATE TABLE tlb (pk int, c
> int, v int, PRIMARY KEY (pk, c)) and the following row: INSERT INTO tlb
> (pk, c, v) VALUES (1, 1, 1)
> deleting the column v (DELETE v FROM %s WHERE pk = 1 AND c = 1) will not
> delete the row as the primary key columns have a timestamp and therefore do
> exist. So the row will still exist with a null value for column v.
>
> If the row was created through an UPDATE (UPDATE tlb SET v = 1 WHERE pk =
> 1 AND c = 1) things will be different as an UPDATE statement do not create
> a timestamp for the clustering columns. By consequence, if V is deleted
> (set to null) the row will not have any columns anymore and will be deleted.
>
> The issue here is that in practice we never consider partition keys or
> clustering columns as null if the database returns a row for it. Whether
> the primary key columns have a timestamp or not.
> I believe that we should ignore that fact too as far as IS NULL/IS NOT
> NULL are concerned. If a row exists, its primary columns should be
> considered as not null. Otherwise we are getting on a really slippery
> slope. The INSERT/UPDATE logic is confusing enough in my opinion without
> adding another layer to it.
>
> One other issue that we have though is that the code for != LWT does not
> work with the Three-Valued logic. If you have: [...] IF col != 1  and col
> is null then in the TVL the value should be UNKNOWN therefore the condition
> should not match.
> It feels to me that we should probably keep that behavior for backward
> compatibility reasons but probably change the behavior in Accord if it is
> not already done.
>
>
>
> Le jeu. 21 mars 2024 à 01:10, German Eichberger via dev <
> dev@cassandra.apache.org> a écrit :
>
>> Hi,
>>
>> +1 I like doing it the SQL way. This makes sense to me.
>>
>> Now, in Cassandra setting a column to null means deleting it and if *all*​
>> columns in a row are null the row is deleted. This might be another edge
>> case...
>>
>> German
>> --
>> *From:* Benjamin Lerer 
>> *Sent:* Wednesday, March 20, 2024 9:15 AM
>> *To:* dev@cassandra.apache.org 
>> *Subject:* [EXTERNAL] [DISCUSS] NULL handling and the unfrozen
>> collection issue
>>
>> You don't often get email from b.le...@gmail.com. Learn why this is
>> important 
>> Hi everybody,
>>
>> CEP-29 (CQL NOT Operator) is hitting the grey area of how we want as a
>> community to handle NULL including for things like unfrozen (multi-cell)
>> collections and I would like to make a proposal for moving forward with
>> NULL related issues.
>>
>> We have currently 2 tickets open about NULL handling (I might have missed
>> others):
>>
>>1. CASSANDRA-10715
>>: Allowing
>>Filtering on NULL
>>2. CASSANDRA-17762
>>: LWT IF col =
>>NULL is inconsistent with SQL NULL
>>
>> We also had previously some discussion on which we touched the subject:
>>
>>- [DISCUSS] LWT UPDATE semantics with + and - when null
>>- CEP-15 multi key transaction syntax
>>
>> In all those tickets and discussions the consensus was to have a behavior
>> similar to SQL.
>>
>> For null comparisons, SQL uses the three-value logic (
>> https://modern-sql.com/concept/three-valued-logic) introducing the need
>> for IS NULL and IS NOT NULL operators. Those conflict with the col = NULL
>> predicate supported in LWT conditions (CASSANDRA-17762
>> ).
>>
>> So far, as Cassandra was only using inclusive operators, comparisons were
>> behaving in an expected way. According to three-valued logic NULL CONTAINS
>> 'foo' should return UNKNOWN and the filtering behavior should exclude
>> everything which is not true.Therefore the row should not be returned 

Re: [DISCUSS] Donating easy-cass-stress to the project

[Caleb Rackliffe]

I do have some familiarity w/ the codebase, and I could help support it in a minor capacity. (Reviews, small fixes, etc.) Probably not something I could spend hours on every week.On Apr 25, 2024, at 5:11 PM, Jon Haddad  wrote:I should probably have noted - since TLP is no more, I renamed tlp-stress to easy-cass-stress around half a year ago when I took it over again.JonOn Thu, Apr 25, 2024 at 3:05 PM Jeff Jirsa  wrote:Unless there’s 2-3 other people who expect to keep working on it, I don’t see how we justify creating a subprojectAnd if there’s not 2-3 people expressing interest, even pulling it into the main project seems riskySo: besides Jon, who in the community expects/desires to maintain this going forward? On Apr 25, 2024, at 5:55 PM, Jon Haddad  wrote:Yeah, I agree with your concerns.  I very firmly prefer a separate subproject.  I've got zero interest in moving from a modern Gradle project to an ant based one.  It would be a lot of work for not much benefit.If we wanted to replace cassandra-stress, I'd say bring in the release artifact as part of the build process instead of tying it all together, but I'm OK if we keep it separate as well.JonOn Thu, Apr 25, 2024 at 2:43 PM Brandon Williams  wrote:I want to begin by saying I am generally +1 on this because I have
become a fan of easy-cass-stress after using it, but I am curious if
this is intended to be a subproject, or replace cassandra-stress?  If
the latter, we are going to have to reconcile the build systems
somehow.  I don't really want to drag ECS back to ant, but I also
don't want two different build systems in-tree.

Kind Regards,
Brandon

On Thu, Apr 25, 2024 at 9:38 AM Jon Haddad  wrote:
>
> I've been asked by quite a few people, both in person and in JIRA [1] about contributing easy-cass-stress [2] to the project.  I've been happy to maintain the project myself over the years but given its widespread use I think it makes sense to make it more widely available and under the project's umbrella.
>
> My goal with the project was always to provide something that's easy to use.  Up and running in a couple minutes, using the parameters to shape the workload rather than defining everything through configuration.  I was happy to make this tradeoff since Cassandra doesn't have very many types of queries and it's worked well for me over the years.
>
> Obviously I would continue working on this project, and I hope this would encourage others to contribute.  I've heard a lot of good ideas that other teams have implemented in their folks.  I'd love to see those ideas make it into the project, and it sounds like it would be a lot easier for teams to get approval to contribute if it was under the project umbrella.
>
> Would love to hear your thoughts.
>
> Thanks,
> Jon
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-18661
> [2] https://github.com/rustyrazorblade/easy-cass-stress

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Caleb Rackliffe]

FYI, there is some ongoing sort-of-related work going on in CASSANDRA-19534
<https://issues.apache.org/jira/browse/CASSANDRA-19534>

On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia 
wrote:

> Just created an official CEP-41
> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
> incorporating the feedback from this discussion. Feel free to let me know
> if I may have missed some important feedback in this thread that is not
> captured in the CEP-41.
>
> Jaydeep
>
> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia <
> chovatia.jayd...@gmail.com> wrote:
>
>> Thanks, Josh. I will file an official CEP with all the details in a few
>> days and update this thread with that CEP number.
>> Thanks a lot everyone for providing valuable insights!
>>
>> Jaydeep
>>
>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie 
>> wrote:
>>
>>> Do folks think we should file an official CEP and take it there?
>>>
>>> +1 here.
>>>
>>> Synthesizing your gdoc, Caleb's work, and the feedback from this thread
>>> into a draft seems like a solid next step.
>>>
>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>
>>> I see a lot of great ideas being discussed or proposed in the past to
>>> cover the most common rate limiter candidate use cases. Do folks think we
>>> should file an official CEP and take it there?
>>>
>>> Jaydeep
>>>
>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe 
>>> wrote:
>>>
>>> I just remembered the other day that I had done a quick writeup on the
>>> state of compaction stress-related throttling in the project:
>>>
>>>
>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>
>>> I'm sure most of it is old news to the people on this thread, but I
>>> figured I'd post it just in case :)
>>>
>>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie 
>>> wrote:
>>>
>>>
>>> 2.) We should make sure the links between the "known" root causes of
>>> cascading failures and the mechanisms we introduce to avoid them remain
>>> very strong.
>>>
>>> Seems to me that our historical strategy was to address individual known
>>> cases one-by-one rather than looking for a more holistic load-balancing and
>>> load-shedding solution. While the engineer in me likes the elegance of a
>>> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me
>>> wonders how far we think we are today from a stable set-point.
>>>
>>> i.e. are we facing a handful of cases where nodes can still get pushed
>>> over and then cascade that we can surgically address, or are we facing a
>>> broader lack of back-pressure that rears its head in different domains
>>> (client -> coordinator, coordinator -> replica, internode with other
>>> operations, etc) at surprising times and should be considered more
>>> holistically?
>>>
>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>
>>> I almost forgot CASSANDRA-15817, which introduced
>>> reject_repair_compaction_threshold, which provides a mechanism to stop
>>> repairs while compaction is underwater.
>>>
>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe 
>>> wrote:
>>>
>>> 
>>> Hey all,
>>>
>>> I'm a bit late to the discussion. I see that we've already discussed
>>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013>
>>>  and CASSANDRA-16663
>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in
>>> passing. Having written the latter, I'd be the first to admit it's a crude
>>> tool, although it's been useful here and there, and provides a couple
>>> primitives that may be useful for future work. As Scott mentions, while it
>>> is configurable at runtime, it is not adaptive, although we did
>>> make configuration easier in CASSANDRA-17423
>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is
>>> global to the node, although we've lightly discussed some ideas around
>>> making it more granular. (For example, keyspace-based limiting, or limiting
>>> "domains" tagged by the client in requests, could be interesting.) It also
>>> does not deal with inter-node traffic, of course.
&g

Re: [DISCUSS] Adding experimental vtables and rules around them

[Caleb Rackliffe]

The two-part proposal of 1.) table-level self-identification of
experimental status and 2.) a global config flag that determines what to do
when querying those might work. I guess the only thing you can't do there
is ignore warnings from a specific experimental table, since that's
controlled in code and not in configuration? You'd probably have to do that
as a custom argument in the native protocol, similar to the client
instructions for what to do on overload.

On Wed, May 29, 2024 at 6:11 PM David Capwell  wrote:

> As another option, we could add an @Experimetal annotation (or another name)
> and a configuration parameter experimental_virtula_tables_enabled
>
>
> Since these are tables we have TableProps so we could always store an
> experimental = true there and if Config.experimental_virtula_tables_enabled
> == false we just skip those tables when we load them.
>
> That way authors would just do the following…
>
> CreateTableStatement.parse(query, keyspace)
>.comment(comment)
>.kind(TableMetadata.Kind.VIRTUAL)
>.experimental(true)
>.build();
>
> And VirtualKeyspace would skip it when 
> Config.experimental_virtula_tables_enabled
> == false
>
> This feels very similar to the JDK… some features require you to opt into
> experimental things twice…
>
> What about putting such a table to system_experimental keyspace?
>
>
> I like it as it makes this apart of its name, but is sad that we must
> break compatibility once we mark it stable (as we move the keyspace).
>
>
> On May 29, 2024, at 1:23 PM, Štefan Miklošovič <
> stefan.mikloso...@gmail.com> wrote:
>
> I think that 2) (client warning on accessing vtable) is pretty obtrusive /
> irritating, that means that every single time I would e.g. SELECT from such
> a table, it would emit a warning to a client / cqlsh? I do not think that
> is necessary.
>
> What about putting such a table to system_experimental keyspace? Then it
> would be about deciding when it is not considered experimental anymore and
> once that happens, we just move it to the appropriate keyspace. I think
> that it is better if users (rather admins) execute a query against
> system_experimental.accord_epochs so they immediately know what they access
> (something experimental) rather than being constantly reminded about that
> by a client warning.
>
> Maxim's idea about "experimental_virtual_tables_enabled" property is also
> viable. However, I can't help it but that somehow reminds me a property for
> a guardrail. Just a thought. I do not see yet how we could leverage it,
> just saying how I perceive it.
>
> On Wed, May 29, 2024 at 9:02 PM David Capwell  wrote:
>
>> We agreed a long time ago that all new features are disabled by default,
>> but I wanted to try to flesh out what we “should” do with something that
>> might still be experimental and subject to breaking changes; I would prefer
>> we keep this thread specific to vtables as the UX is different for
>> different types of things…
>>
>> So, lets say we are adding a set of vtables but we are not 100% sure what
>> the schema should be and we learn after the release that changes should be
>> made, but that would end up breaking the table… we currently define
>> everything as “don’t break this” so if we publish a table that isn’t 100%
>> baked we are kinda stuck with it for a very very long time… I would like to
>> define a way to expose vtables that are subject to change (breaking schema
>> changes) across different release and rules around them (only in minor?
>> Maybe even in patch?).
>>
>> Lets try to use a concrete example so everyone is on the same page.
>>
>> Accord is disabled by default (it is a new feature), so the vtables to
>> expose internals would be expected to be undefined and not present on the
>> instance.
>>
>> When accord is enabled (accord.enabled = true) we add a set of vtables:
>>
>> Epochs - shows what epochs are known to accord
>> Cache - shows how the internal caches are performing
>> Etc.
>>
>> Using epochs as an example it currently only shows a single column: the
>> long epoch
>>
>> CREATE VIRTUAL TABLE system_accord.epochs (epoch bigint PRIMARY KEY);
>>
>> Lets say we find that this table isn’t enough and we really need to scope
>> it to each of the “stores” (threads for processing accord tasks)
>>
>> CREATE VIRTUAL TABLE system_accord.epochs (epoch bigint, store_id int,
>> PRIMARY KEY (epoch, store_id));
>>
>> In this example the table changed the schema in a way that could break
>> users, so this normally is not allowed.
>>
>> Since we don’t really have a way to define something experimental other
>> than NEWS.txt, we kinda get stuck with this table and are forced to make
>> new versions and maintain them for a long time (in this example we would
>> have epochs and epochs_v2)… it would be nice if we could define a way to
>> express that tables are free to be changed (modified or even deleted) and

Re: [DISCUSS] Stream Pipelines on hot paths

[Caleb Rackliffe]

+1

On Thu, May 30, 2024 at 11:29 AM Benedict  wrote:

> Since it’s related to the logging discussion we’re already having, I have
> seen stream pipelines showing up in a lot of traces recently. I am
> surprised; I thought it was understood that they shouldn’t be used on hot
> paths as they are not typically as efficient as old skool for-each
> constructions done sensibly, especially for small collections that may
> normally take zero or one items.
>
> I would like to propose forbidding the use of streams on hot paths without
> good justification that the cost:benefit is justified.
>
> It looks like it was nominally agreed two years ago that we would include
> words to this effect in the code style guide, but I forgot to include them
> when I transferred the new contents from the Google Doc proposal. So we
> could just include the “Performance” section that was meant to be included
> at the time.
>
> lists.apache.org
> 
> [image: favicon.ico]
> 
> 
>
>
> On 30 May 2024, at 13:33, Štefan Miklošovič 
> wrote:
>
> 
> I see the feedback is overall positive. I will merge that and I will
> improve the documentation on the website along with what Benedict suggested.
>
> On Thu, May 30, 2024 at 10:32 AM Mick Semb Wever  wrote:
>
>>
>>
>>
>>> Based on these findings, I went through the code and I have incorporated
>>> these rules and I rewrote it like this:
>>>
>>> 1) no wrapping in "if" if we are not logging more than 2 parameters.
>>> 2) rewritten log messages to not contain any string concatenation but
>>> moving it all to placeholders ({}).
>>> 3) wrap it in "if" if we need to execute a method(s) on parameter(s)
>>> which is resource-consuming.
>>>
>>
>>
>> +1
>>
>>
>> It's a shame slf4j botched it with lambdas, their 2.0 fluent api doesn't
>> impress me.
>>
>


favicon.ico
Description: Binary data

[DISCUSS] Increments on non-existent rows in Accord

[Caleb Rackliffe]

We had a bug report a while back from Luis E Fernandez and team in
CASSANDRA-18988 
around the behavior of increments/decrements on numeric fields for
non-existent rows. Consider the following, wich can be run on the
cep-15-accord branch:

CREATE KEYSPACE accord WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '1'} AND durable_writes = true


CREATE TABLE accord.accounts (
partition text,
account_id int,
balance int,
PRIMARY KEY (partition, account_id)
) WITH CLUSTERING ORDER BY (account_id ASC) AND transactional_mode='full'


BEGIN TRANSACTION
INSERT INTO accord.accounts (partition, account_id, balance)
VALUES ('default', 0, 100);
INSERT INTO accord.accounts (partition, account_id, balance)
VALUES ('default', 1, 100);
COMMIT TRANSACTION


BEGIN TRANSACTION
UPDATE accord.accounts SET balance -= 10 WHERE partition =
'default' AND account_id = 1;
UPDATE accord.accounts SET balance += 10 WHERE partition =
'default' AND account_id = 3;
COMMIT TRANSACTION


Reading the 'default' partition will produce the following result.


 partition | account_id | balance
---++-
   default |  0 | 100
   default |  1 |  90


As you will notice, we have not implicitly inserted a row for
account_id 3, which does not exist when we request that its balance be
incremented by 10. This is by design, as null + 10 == null.


Before I close CASSANDRA-18988
, *I'd like to
confirm with everyone reading this that the behavior above is
reasonable*. The only other option I've seen proposed that would make
sense is perhaps producing a result like:


 partition | account_id | balance
---++-
   default |  0 | 100
   default |  1 |  90

   default |  3 |null



Note however that this is exactly what we would produce if we had
first inserted a row w/ no value for balance:


INSERT INTO accord.accounts (partition, account_id) VALUES ('default', 3);