Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Štefan Miklošovič]

That is all OK to mention. So as I read it correctly, Jon, myself, Andrew,
David - we all would like to see some cross-language events ingestion.
Other people do not seem to consider it important enough (just correct me
if I am wrong). I do not mind, I am 50:50, not an absolute must but sure,
let's add that to the wish list.

I believe that unless it is an absolute show-stopper (which it is not) it
is not necessary. It is something to keep in mind upon next
refactorization. I am personally not affected by this. Whoever will need
that, they will find a way to make it happen.

I would really appreciate it if we found consensus among these points I
wrote down. From my point of view, the most ideal outcome would be to make
CEP-12 happen which cleans it all up and makes it more robust.

My perception is that we have always found the most practical solution and
if it brings value to a user (being able to inspect diagnostic events
offline for further inspection) we should not avoid delivering that.
Something similar happened in e.g. Password validator / generator (CEP-24)
where the most ideal solution was to base it on transactional configuration
but even though we are not there yet, that did not stop us from delivering
it because for some entities in this space it brought value anyway.

It may seem as if I am invested in delivering it because I spent some time
on that already - that is not the case - I am OK to drop diagnostic events
persistence if you insist on that but honestly I just do not see any
compelling argument to do so. The library we are using that for (Chronicle
Queues) is already there, it is just a functionality addition and as I
said, if somebody comes and delivers a solution which would replace it, I
do not see it to be problematic. It would be replaced as anything else
(FQL, Audit ...).

On Sun, Sep 29, 2024 at 9:06 PM Jon Haddad  wrote:

> Strong +1 to the file format issue, and if we're building a wish list - it
> would be great if we could read the file format without pulling in
> cassandra-all.  Long term, I'd love to see this for SSTables & Commit logs
> as well.
>
> I've long been a fan of Gradle subprojects because it makes this kind of
> thing fairly easy.
>
> Jon
>
>
>
> On Sun, Sep 29, 2024 at 11:46 AM Andrew Weaver 
> wrote:
>
>> I'm late to the discussion here, but I want to add my experience from
>> dealing with audit logs specifically.
>>
>> Chronicle has some advantages (binary, compact) but it has a serious
>> disadvantage from a consumption standpoint. It's not a well-supported file
>> format. Audit logs are something that I think most operators are interested
>> in archiving for compliance purposes and analyzing offline for any number
>> of reasons and an oddball file format is an unnecessary hurdle for the
>> audit logs use-case.
>>
>> I would welcome support for an existing format that is compact,
>> high-performance and compatible with common tools (Spark, etc.).
>>
>> On Sun, Sep 29, 2024, 10:11 AM Štefan Miklošovič 
>> wrote:
>>
>>> Thank you all for your answers and opinions. I would like to have some
>>> kind of a resolution here in order to move forward, especially with
>>> relation to CEP-12 I mentioned earlier. (1).
>>>
>>> I think we have these options:
>>>
>>> 1) Do nothing and wait until this gets back to us, probably in a more
>>> serious way (we find a bug and we will not be able to update it because it
>>> would be on "ea" or new features will be available only in newer versions).
>>>
>>> 2) Fork it and continue to maintain it - I do not think this is
>>> realistic, nobody is going to take care of forking that and maintaining it
>>> long term.
>>>
>>> 3) Do nothing but refactor it in such a way that it will be easier to
>>> replace it with something else in the future. CEP-12 is not only adding
>>> persistence to diagnostic events but the patch I have also makes whole
>>> logging more robust. Even it is all on Chronicle Queues (FQL, Audit ...),
>>> there are some differences between that when it comes to the implementation
>>> and I think that refactoring it in such a way that it would have all clear
>>> class structure and hierarchy (bottom of CEP-12) we will have easier job if
>>> we ever go to replace that.
>>>
>>> 4) Proceed with CEP-12 even though we know we are building it on top of
>>> something which should not be there.
>>>
>>> 5) Do absolutely nothing until we replace it with something else and we
>>> get rid of what is there right now - that would mean that we will not
>>> benefit from the code which is easier to maintain etc (if CEP-12 is not
>>> going to materialize) which I think is a welcomed attribute of the code
>>> base to have.
>>>
>>> I was thinking more about stuff like protobuf and while I do see
>>> benefits of that, honestly, it just does not matter too much if it is done
>>> like that or not. I mean, sure, it would be cool to have, but we could
>>> spend a lot of effort on protobuf and integrating with it or on anything
>>> which would 

Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

[Dmytro Tsapko]

Hello, LGTM.

On Thu, Sep 26, 2024 at 11:46 PM Bret McGuire 
wrote:

>Greetings all!
>
>I’m proposing the Cassandra Gocql Driver 1.7.0 for release.
>
> sha1: 953e0df999cabb3f5eef714df9921c00e9f632c2
> git: https://github.com/apache/cassandra-gocql-driver/tree/v1.7.0-rc1
>
>The Source release is available here:
>
> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-gocql-driver/1.7.0/
>
>This is the first release of the Gocql Driver since its donation.
> Developers will include this driver in their projects by specifying a
> commit hash or tag in go.mod rather than via the inclusion of binary
> artifacts so we’ve avoided the creation of binary artifacts completely.  To
> avoid any premature access to this release before the vote is complete
> we’ve temporarily used the tag “v1.7.0-rc1” to clearly indicate that this
> tag points to a release candidate.  Once the vote passes this tag will be
> updated to “v1.7.0”.
>
>The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
>
>Thanks all!
>
>   - Bret -
>

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Benedict]

Yes, with accord it should be fairly easy to have reliable no-dupe log streaming without an elected leader. Given the broad set of use cases, I can imagine supporting some more native topic subscription API in C* rather than requiring Kafka, so perhaps any integration of Kafka with the sidecar can be considered a separate parallel effort, that might eventually implement itself with this C* feature whenever it materialises?On 30 Sep 2024, at 03:42, Jeff Jirsa  wrote:Transactional metadata and Accord should make it MUCH easier to do duplication avoiding CDC (and I was going to note that someone should ensure that the interfaces exposed to the public are stable enough not to change the published api once those exist)On Sep 29, 2024, at 7:04 PM, Patrick McFadin  wrote:As I was reviewing this, it occurred to me that it was talking about Sidecar like it was a thing but that CEP has been stalled for quite some time:  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224If work on this is being done, should we get this official and wrapped up?On to the proposal...This has been a topic on the project for over 10 years now. I've seen multiple goes at making this work and the issue that always turns out to torpedo the project is handing dupes. To the point that they go from a generalized Kafka producer engine to something specific to a particular use case. I don't see much on how this would be handled other than "left to the end user to figure out." There is also little mention of where the increased resource load would be handled. This has been discussed many times before, but is it time to introduce the concept of an elected leader for a token range for this type of operation? It would eliminate a ton of problems that need to managed when bridging c* to a system like Kafka. Last time it was discussed in earnest was for KIP-30: https://cwiki.apache.org/confluence/display/KAFKA/KIP-30+-+Allow+for+brokers+to+have+plug-able+consensus+and+meta+data+storage+sub+systems PatrickOn Sat, Sep 28, 2024 at 11:44 AM Jon Haddad  wrote:Yes! I’m really looking forward to trying this out. The CEP looks really well thought out. I think this will make CDC a lot more useful for a lot of teams. JonOn Fri, Sep 27, 2024 at 4:23 PM Josh McKenzie  wrote:Really excited to see this hit the ML James.As author of the base CDC (get your stones ready for throwing :D) and someone moderately involved in the CEP here, definitely welcome any questions. CDC is a thorny problem in a multi-replica distributed system like this.On Fri, Sep 27, 2024, at 5:40 PM, James Berragan wrote:Hi everyone,Wiki: https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-44%3A+Kafka+integration+for+Cassandra+CDC+using+SidecarWe would like to propose this CEP for adoption by the community.CDC is a common technique in databases but right now there is no out-of-the-box solution to do this easily and at scale with Cassandra. Our proposal is to build a fully-fledged solution into the Apache Cassandra Sidecar. This comes with a number of benefits:- Sidecar is an official part of the existing Cassandra eco-system.- Sidecar runs co-located with Cassandra instances and so scales with the cluster size.- Sidecar can access the underlying Cassandra database to store CDC configuration and the CDC state in a special table.- Running in the Sidecar does not require additional external resources to run.The core CDC module we anticipate will be pluggable and re-usable, it is available for review here: https://github.com/apache/cassandra-analytics/pull/87. The remaining Sidecar code will follow.As a reminder, please keep the discussion here on the dev list vs. in the wiki, as we’ve found it easier to manage via email.Sincerely,James BerraganBernardo Botella CorbiYifan CaiJyothsna Konisa

Re: CEP-15: Accord status

[Štefan Miklošovič]

I think there was some discussion about putting that all to the website
which never materialized.

On Mon, Sep 30, 2024 at 2:37 PM Josh McKenzie  wrote:

> Today I learned… I had no clue we had markdown files in src/java…
>
> Discoverability issues in our codebase?
>
> Well I never.
>
> ;)
>
> On Sat, Sep 28, 2024, at 10:39 PM, David Capwell wrote:
>
> Today I learned… I had no clue we had markdown files in src/java…
>
> $ find src/ -name '*.md'
> src//java/org/apache/cassandra/io/sstable/SSTable_API.md
> src//java/org/apache/cassandra/io/sstable/format/bti/BtiFormat.md
> src//java/org/apache/cassandra/utils/bytecomparable/ByteComparable.md
> src//java/org/apache/cassandra/tcm/TCM_implementation.md
> src//java/org/apache/cassandra/tcm/TransactionalClusterMetadata.md
> src//java/org/apache/cassandra/db/memtable/Memtable_API.md
> src//java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.md
> src//java/org/apache/cassandra/db/tries/InMemoryTrie.md
> src//java/org/apache/cassandra/db/tries/Trie.md
> src//java/org/apache/cassandra/index/sai/README.md
> src//java/org/apache/cassandra/service/paxos/Paxos.md
>
>
>
> We don’t have one at the moment but it would be good to get that in.  At a
> high level there are a few key classes
>
> 1) org.apache.cassandra.cql3.statements.TransactionStatement - this class
> handles BEGIN TRANSACTION in CQL
> 2) org.apache.cassandra.service.consensus.TransactionalMode - this is a
> table property and dictates what is allowed for the table.  If off accord
> transactions are not allowed, if “full” normal read/write get migrated to
> Accord (and you can still use BEGIN TRANSACTION)
> 3) org.apache.cassandra.service.accord.AccordService - the global static
> instance that lets Cassandra call Accord stuff
>
> On Sep 27, 2024, at 7:20 AM, Paulo Motta  wrote:
>
> Thanks all for the work on this epic!
>
> Is there an implementation summary guide similar to guide_8099.md [1] that
> can help reviewers not involved with the effort navigate through the code ?
> It would be great to have it if this is not already available or being
> planned. There's a similar one though much smaller in scope for memtable
> API on [2].
>
> [1] -
> https://github.com/apache/cassandra/blob/cassandra-3.0.0-rc2/guide_8099.md
>
> [2] -
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
>
> On Fri, Sep 27, 2024 at 8:09 AM Benedict Elliott Smith <
> bened...@apache.org> wrote:
>
> If you exclude test changes, there’s < 50k added and ~2k removed. This
> works out to ~7% of the scale of 8099 for lines modified, if this is the
> benchmark for disruption.
>
> Altogether, this is a very small patch from the perspective of the
> existing codebase. Probably doesn’t even come close to the top 10.
>
> Conversely, for new standalone features, this is likely the most complex
> thing we have ever merged to the project. But, it is off by default, and
> the risk to deployments therefore is very minimal.
>
> Regarding how parties can engage, I think if we’re honest history shows
> that engagement will be minimal. There have after all been several touch
> points, and none have materialised into really significant engagement. This
> is just the reality of everyone having their own pressures - at the end of
> the day, changes happen and the community adapts. But, we are here to
> answer any questions - as we have been throughout the development of the
> work in the open.
>
>
>
> On 20 Sep 2024, at 22:08, Josh McKenzie  wrote:
>
> This presents an opportune moment for those interested to review the code.
> ...
> +88,341 −7,341
> 1003 Files changed
>
>
> O.o
> This is... *very large*. If we use CASSANDRA-8099 as our "banana for
> scale":
>
> 645 files changed, 49381 insertions(+), 42227 deletions(-)
>
>
> To be clear - I don't think we collectively should be worried about
> disruption from this patch since:
>
>1. Each commit (or the vast majority?) has already been reviewed by >=
>1 other committer
>2. 7.3k deletions is a lot less than 42k
>3. We now have fuzzing, property based testing, and the simulator
>4. Most of this code is additive
>
> How would you recommend interested parties engage with reviewing this
> behemoth? Or perhaps subsections of it or key areas to familiarize
> themselves with the structure?
>
> On Fri, Sep 20, 2024, at 12:17 PM, David Capwell wrote:
>
> Recently, we rebased against the trunk branch, ensuring that the accord
> branch is now in sync with the latest trunk version. This presents an
> opportune moment for those interested to review the code.
>
> We have a pending pull request (*https://github.com/apache/cassandra/pull/3552
> *) that we do not intend
> to merge.
>
> Our current focus is on addressing several bug fixes and ensuring the
> safety of topology changes (as evidenced by the number of issues filed
> against the trunk). Once we wrap up bug fixes and safety feature

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Josh McKenzie]

> I don't see much on how this would be handled other than "left to the end 
> user to figure out." 
My immediate thought when I read that was "Yes. But it's moving where we draw 
the line of 'left to the end user to figure out' *much further* than it was 
before".

This should only be necessary in edge cases w/extended severe degraded 
availability where you can't hit QUORUM w/this design. So we go from "De-dupe 
literally everything o ye' user" to "de-dupe a small fraction of a % of the 
time when things really go off the rails".

It still leaves the burden of processing potential duplicates downstream, so 
some *complexity* burden on the users remains if they have no tolerance for 
processing duplicate messages, however the underlying machine resource 
utilization (from "dedupe everything" to "dedupe a small % of things") is 
pretty massively shifted by this design change. That, and using the hash of the 
mutation the way the extended design does is something a downstream consumer 
could also do on their side to ensure anything that came in past the drop-off 
window wasn't already seen. So not *too* painful; certainly a vast improvement 
over the status quo.

As to TCM and Accord: absolutely agree. I'd love to see a world where we Accord 
everything and fire off CDC to subscribers from a coordinator bypassing all 
this LSM-bastardized post-processing for CDC for instance. That said, this is a 
functionality users needed back in... 2016? When we first added CDC. So I think 
it's worth it to move on it now while retaining architectural options to move 
to updated metadata and transactions as they mature (obviously we'll lean on 
TCM since it's in 5.0 / trunk right now; more applies to the accord bit).

On Mon, Sep 30, 2024, at 3:20 AM, Benedict wrote:
> 
> Yes, with accord it should be fairly easy to have reliable no-dupe log 
> streaming without an elected leader. Given the broad set of use cases, I can 
> imagine supporting some more native topic subscription API in C* rather than 
> requiring Kafka, so perhaps any integration of Kafka with the sidecar can be 
> considered a separate parallel effort, that might eventually implement itself 
> with this C* feature whenever it materialises?
> 
> 
>> On 30 Sep 2024, at 03:42, Jeff Jirsa  wrote:
>> 
>> 
>> Transactional metadata and Accord should make it MUCH easier to do 
>> duplication avoiding CDC (and I was going to note that someone should ensure 
>> that the interfaces exposed to the public are stable enough not to change 
>> the published api once those exist)
>> 
>> 
>> 
>>> On Sep 29, 2024, at 7:04 PM, Patrick McFadin  wrote:
>>> 
>>> As I was reviewing this, it occurred to me that it was talking about 
>>> Sidecar like it was a thing but that CEP has been stalled for quite some 
>>> time:  
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
>>> 
>>> If work on this is being done, should we get this official and wrapped up?
>>> 
>>> On to the proposal...
>>> 
>>> This has been a topic on the project for over 10 years now. I've seen 
>>> multiple goes at making this work and the issue that always turns out to 
>>> torpedo the project is handing dupes. To the point that they go from a 
>>> generalized Kafka producer engine to something specific to a particular use 
>>> case. I don't see much on how this would be handled other than "left to the 
>>> end user to figure out." 
>>> 
>>> There is also little mention of where the increased resource load would be 
>>> handled. 
>>> 
>>> This has been discussed many times before, but is it time to introduce the 
>>> concept of an elected leader for a token range for this type of operation? 
>>> It would eliminate a ton of problems that need to managed when bridging c* 
>>> to a system like Kafka. Last time it was discussed in earnest was for 
>>> KIP-30: 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-30+-+Allow+for+brokers+to+have+plug-able+consensus+and+meta+data+storage+sub+systems
>>>  
>>> 
>>> Patrick
>>> 
>>> On Sat, Sep 28, 2024 at 11:44 AM Jon Haddad  
>>> wrote:
 Yes! I’m really looking forward to trying this out. The CEP looks really 
 well thought out. I think this will make CDC a lot more useful for a lot 
 of teams. 
 Jon
 
 
 On Fri, Sep 27, 2024 at 4:23 PM Josh McKenzie  wrote:
> __
> Really excited to see this hit the ML James.
> 
> As author of the base CDC (get your stones ready for throwing :D) and 
> someone moderately involved in the CEP here, definitely welcome any 
> questions. CDC is a *thorny* *problem *in a multi-replica distributed 
> system like this.
> 
> On Fri, Sep 27, 2024, at 5:40 PM, James Berragan wrote:
>> Hi everyone,
>> 
>> Wiki: 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-44%3A+Kafka+integration+for+Cassandra+CDC+using+Sidecar
>> 
>> We would like to propose this CEP for adoption by the community.
>> 
>> CDC is a commo

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Patrick McFadin]

I made the mistake of asking two things in one email.

First thing I asked. Sidecar? Stalled CEP so why is this being talked about
like this is a thing?

On Mon, Sep 30, 2024 at 7:21 AM Benedict  wrote:

> Sorry Bernardo, you may have misunderstood me. I don’t have any concerns,
> I was suggesting a possible future scenario where CDC for Kafka via sidecar
> is changed to use a hypothetical future topic subscription service provided
> by C*. It was meant to show that this CEP may be easily decoupled from any
> future evolution in this area.
>
> On 30 Sep 2024, at 14:58, Bernardo Botella 
> wrote:
>
> Thanks everyone for the comments.
>
>
> Patrick:
> The proposal includes a “best effort” approach for deduplication (some
> details can be found on the Digest class comments on the PR here
> https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
>  ).
> That alone won’t eliminate all the duplicates, but as Josh points out, it
> moves the line to something way easier to handle for consumers, and
> definitely on the direction we should aim towards. Accord is definitely
> something this contribution will benefit from, that will move that line
> even further.
>
> Benedict:
> If I understand it correctly, your concern is that Kafka is somewhat the
> hardcoded option for a CDC stream being published? The proposal introduces
> a concept of data sources and sinks (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks)
> being kafka the first implemented data sink. That means that the actual
> Kafka output should (will) be something pluggable.
>
>
>
> On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:
>
> I don't see much on how this would be handled other than "left to the end
> user to figure out."
>
> My immediate thought when I read that was "Yes. But it's moving where we
> draw the line of 'left to the end user to figure out' *much further* than
> it was before".
>
> This should only be necessary in edge cases w/extended severe degraded
> availability where you can't hit QUORUM w/this design. So we go from
> "De-dupe literally everything o ye' user" to "de-dupe a small fraction of a
> % of the time when things really go off the rails".
>
> It still leaves the burden of processing potential duplicates downstream,
> so some *complexity* burden on the users remains if they have no
> tolerance for processing duplicate messages, however the underlying machine
> resource utilization (from "dedupe everything" to "dedupe a small % of
> things") is pretty massively shifted by this design change. That, and using
> the hash of the mutation the way the extended design does is something a
> downstream consumer could also do on their side to ensure anything that
> came in past the drop-off window wasn't already seen. So not *too* painful;
> certainly a vast improvement over the status quo.
>
> As to TCM and Accord: absolutely agree. I'd love to see a world where we
> Accord everything and fire off CDC to subscribers from a coordinator
> bypassing all this LSM-bastardized post-processing for CDC for instance.
> That said, this is a functionality users needed back in... 2016? When we
> first added CDC. So I think it's worth it to move on it now while retaining
> architectural options to move to updated metadata and transactions as they
> mature (obviously we'll lean on TCM since it's in 5.0 / trunk right now;
> more applies to the accord bit).
>
> On Mon, Sep 30, 2024, at 3:20 AM, Benedict wrote:
>
>
> Yes, with accord it should be fairly easy to have reliable no-dupe log
> streaming without an elected leader. Given the broad set of use cases, I
> can imagine supporting some more native topic subscription API in C* rather
> than requiring Kafka, so perhaps any integration of Kafka with the sidecar
> can be considered a separate parallel effort, that might eventually
> implement itself with this C* feature whenever it materialises?
>
>
> On 30 Sep 2024, at 03:42, Jeff Jirsa  wrote:
>
> 
>
> Transactional metadata and Accord should make it MUCH easier to do
> duplication avoiding CDC (and I was going to note that someone should
> ensure that the interfaces exposed to the public are stable enough not to
> change the published api once those exist)
>
>
>
> On Sep 29, 2024, at 7:04 PM, Patrick McFadin  wrote:
>
> 
> As I was reviewing this, it occurred to me that it was talking about
> Sidecar like it was a thing but that CEP has been stalled for quite some
> time:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
>
> If work on this is being done, should we get this official and wrapped up?
>
> On to the proposal...
>
> This has been a topic on the project for over 10 years now. I've seen
> multiple goes at making this work and the issue that always turns out to
> torpedo the project is handing dupes. To the point that they go from a
>

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Dinesh Joshi]

Patrick, could you please elaborate? The Sidecar has been a thing for a
while now.

On Mon, Sep 30, 2024 at 7:51 AM Patrick McFadin  wrote:

> I made the mistake of asking two things in one email.
>
> First thing I asked. Sidecar? Stalled CEP so why is this being talked
> about like this is a thing?
>
> On Mon, Sep 30, 2024 at 7:21 AM Benedict  wrote:
>
>> Sorry Bernardo, you may have misunderstood me. I don’t have any concerns,
>> I was suggesting a possible future scenario where CDC for Kafka via sidecar
>> is changed to use a hypothetical future topic subscription service provided
>> by C*. It was meant to show that this CEP may be easily decoupled from any
>> future evolution in this area.
>>
>> On 30 Sep 2024, at 14:58, Bernardo Botella 
>> wrote:
>>
>> Thanks everyone for the comments.
>>
>>
>> Patrick:
>> The proposal includes a “best effort” approach for deduplication (some
>> details can be found on the Digest class comments on the PR here
>> https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
>>  ).
>> That alone won’t eliminate all the duplicates, but as Josh points out, it
>> moves the line to something way easier to handle for consumers, and
>> definitely on the direction we should aim towards. Accord is definitely
>> something this contribution will benefit from, that will move that line
>> even further.
>>
>> Benedict:
>> If I understand it correctly, your concern is that Kafka is somewhat the
>> hardcoded option for a CDC stream being published? The proposal introduces
>> a concept of data sources and sinks (
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks)
>> being kafka the first implemented data sink. That means that the actual
>> Kafka output should (will) be something pluggable.
>>
>>
>>
>> On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:
>>
>> I don't see much on how this would be handled other than "left to the end
>> user to figure out."
>>
>> My immediate thought when I read that was "Yes. But it's moving where we
>> draw the line of 'left to the end user to figure out' *much further* than
>> it was before".
>>
>> This should only be necessary in edge cases w/extended severe degraded
>> availability where you can't hit QUORUM w/this design. So we go from
>> "De-dupe literally everything o ye' user" to "de-dupe a small fraction of a
>> % of the time when things really go off the rails".
>>
>> It still leaves the burden of processing potential duplicates downstream,
>> so some *complexity* burden on the users remains if they have no
>> tolerance for processing duplicate messages, however the underlying machine
>> resource utilization (from "dedupe everything" to "dedupe a small % of
>> things") is pretty massively shifted by this design change. That, and using
>> the hash of the mutation the way the extended design does is something a
>> downstream consumer could also do on their side to ensure anything that
>> came in past the drop-off window wasn't already seen. So not *too* painful;
>> certainly a vast improvement over the status quo.
>>
>> As to TCM and Accord: absolutely agree. I'd love to see a world where we
>> Accord everything and fire off CDC to subscribers from a coordinator
>> bypassing all this LSM-bastardized post-processing for CDC for instance.
>> That said, this is a functionality users needed back in... 2016? When we
>> first added CDC. So I think it's worth it to move on it now while retaining
>> architectural options to move to updated metadata and transactions as they
>> mature (obviously we'll lean on TCM since it's in 5.0 / trunk right now;
>> more applies to the accord bit).
>>
>> On Mon, Sep 30, 2024, at 3:20 AM, Benedict wrote:
>>
>>
>> Yes, with accord it should be fairly easy to have reliable no-dupe log
>> streaming without an elected leader. Given the broad set of use cases, I
>> can imagine supporting some more native topic subscription API in C* rather
>> than requiring Kafka, so perhaps any integration of Kafka with the sidecar
>> can be considered a separate parallel effort, that might eventually
>> implement itself with this C* feature whenever it materialises?
>>
>>
>> On 30 Sep 2024, at 03:42, Jeff Jirsa  wrote:
>>
>> 
>>
>> Transactional metadata and Accord should make it MUCH easier to do
>> duplication avoiding CDC (and I was going to note that someone should
>> ensure that the interfaces exposed to the public are stable enough not to
>> change the published api once those exist)
>>
>>
>>
>> On Sep 29, 2024, at 7:04 PM, Patrick McFadin  wrote:
>>
>> 
>> As I was reviewing this, it occurred to me that it was talking about
>> Sidecar like it was a thing but that CEP has been stalled for quite some
>> time:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
>>
>> If work on this is being done, should we get this official and wrapped up?

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[James Berragan]

Thanks for the discussions. I do anticipate that Accord will make things
very much better, however I think if consumers are ultimately going to be
replay the log into some other system (say Apache Iceberg) exact-once
delivery will always be tricky, but perhaps not entirely necessary given
the linearizability guarantees Accord will bring, either to safely replay
mutations in order or to do something like "APPLY mutation IFF
current.txn_id < mutation.txn_id". This does still mean some load on the
end user to verify, even if the guarantees are stronger, and not too far
off what we currently recommend to verify the last-write-win timestamps.
(My Accord insight is obviously limited so I'm happy to be corrected if I'm
wrong anywhere here!)

On token leadership, the Sidecar comes with soft/implicit primary range
ownership of the co-located Cassandra node(s), we implemented a basic
failover process to satisfy availability upto RF=3, but I think for strong
guarantees hooking into TCM would be the ultimate goal.

James.

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Josh McKenzie]

I say we do CEP-12 w/out using chronicle and have a follow up JIRA to replace 
chronicle w/something else.

Seems like there's a reasonable consensus around at least that subset of things 
given the discussion here. As to what form that something else takes, well, 
that's a topic for another day. :)

On Mon, Sep 30, 2024, at 10:09 AM, Štefan Miklošovič wrote:
>  "how complex should it be to rip out the chronicle format, insert some other 
> well defined and well known, and handle log rolling ourselves".
>  
> I think that it will be actually easier to do after CEP-12 is in because as I 
> mentioned it does housekeeping of what is there rigth now and refactors it a 
> little bit so throwing away the guts of it should be isolated only to actual 
> BinLog class and stuff around that which is built on top of Chronicle.
> 
> "There a reason we can't move forward with CEP-12 w/out addressing the 
> chronicle stuff? i.e."
> 
> I think there is not any.
> 
> "Why would CEP-12 be heavily coupled with chronicle?" - it would not be 
> heavier from what is already there for audit of fql. Chronicle here basically 
> acts as a sink. 
> 
> Actually, that patch also makes the implementations of (diagnostic too) 
> loggers pluggable  (via coding against an interface and putting that on the 
> class path) so people might already write it to whatever they want - even to 
> something protobuf-like. If they do not want to use Chronicle as a sink, by 
> implementing their own logger, they could just put it wherever they want. I 
> think that I forgot to mention this aspect. So the whole solution we have is 
> already not hard-wired to Chronicle necessarily. It is just something we 
> provide out of the box.
> 
> On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie  wrote:
>> __
>> I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking 
>> through "how complex should it be to rip out the chronicle format, insert 
>> some other well defined and well known, and handle log rolling ourselves". 
>> My preference (which I didn't indicate earlier) would be to have that done 
>> independently.
>> 
>> There a reason we can't move forward with CEP-12 w/out addressing the 
>> chronicle stuff? i.e.
>>> I would like to have this resolved because there is CEP-12 I plan to 
>>> deliver and I hit this and I do not want to base that work on something we 
>>> might eventually abandon.
>> Why would CEP-12 be heavily coupled with chronicle? I would assume you'd be 
>> able to make light use of the existing logging + log rolling, and then 
>> someone else could come along and easily rip out chronicle and the rolling 
>> and add in a different one with minimal code changes?
>> 
>> On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
>>> I don't understand why CEP-12 should be a vehicle for introducing changes 
>>> like that. That is something totally unrelated. I am not going to be the 
>>> one to implement anything beyond CEP-12 and I am not the one who is going 
>>> to replace it either so if we make it a hard requirement for CEP-12 then 
>>> CEP-12 as it is will never be introduced. Just want to be clear about that.
>>> 
>>> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
 On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  wrote:
 >
 > I'd strongly support either rolling the format change into the CEP-12 
 > proposal or having another CEP for introducing protobuf, spark, etc - 
 > some kind of more broadly adopted format, and removing chronicle from 
 > our stack.
 
 +1, I too would strongly support an open format and removing chronicle.
 
 Kind Regards,
 Brandon
>> 

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Josh McKenzie]

The CEP for the sidecar has stalled. The sidecar itself is very much alive and 
a thing.

CEP != artifact.

We should definitely clean that up though.

On Mon, Sep 30, 2024, at 10:59 AM, Dinesh Joshi wrote:
> Patrick, could you please elaborate? The Sidecar has been a thing for a while 
> now.
> 
> On Mon, Sep 30, 2024 at 7:51 AM Patrick McFadin  wrote:
>> I made the mistake of asking two things in one email. 
>> 
>> First thing I asked. Sidecar? Stalled CEP so why is this being talked about 
>> like this is a thing?
>> 
>> On Mon, Sep 30, 2024 at 7:21 AM Benedict  wrote:
>>> 
>>> Sorry Bernardo, you may have misunderstood me. I don’t have any concerns, I 
>>> was suggesting a possible future scenario where CDC for Kafka via sidecar 
>>> is changed to use a hypothetical future topic subscription service provided 
>>> by C*. It was meant to show that this CEP may be easily decoupled from any 
>>> future evolution in this area. 
>>> 
>>> 
 On 30 Sep 2024, at 14:58, Bernardo Botella  
 wrote:
 Thanks everyone for the comments.
 
 Patrick:
 The proposal includes a “best effort” approach for deduplication (some 
 details can be found on the Digest class comments on the PR here 
 https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
  ). That alone won’t eliminate all the duplicates, but as Josh points out, 
 it moves the line to something way easier to handle for consumers, and 
 definitely on the direction we should aim towards. Accord is definitely 
 something this contribution will benefit from, that will move that line 
 even further.
 
 Benedict:
 If I understand it correctly, your concern is that Kafka is somewhat the 
 hardcoded option for a CDC stream being published? The proposal introduces 
 a concept of data sources and sinks 
 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks)
  being kafka the first implemented data sink. That means that the actual 
 Kafka output should (will) be something pluggable.
 
 
 
> On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:
> 
>> I don't see much on how this would be handled other than "left to the 
>> end user to figure out." 
> My immediate thought when I read that was "Yes. But it's moving where we 
> draw the line of 'left to the end user to figure out' *much further* than 
> it was before".
> 
> This should only be necessary in edge cases w/extended severe degraded 
> availability where you can't hit QUORUM w/this design. So we go from 
> "De-dupe literally everything o ye' user" to "de-dupe a small fraction of 
> a % of the time when things really go off the rails".
> 
> It still leaves the burden of processing potential duplicates downstream, 
> so some *complexity* burden on the users remains if they have no 
> tolerance for processing duplicate messages, however the underlying 
> machine resource utilization (from "dedupe everything" to "dedupe a small 
> % of things") is pretty massively shifted by this design change. That, 
> and using the hash of the mutation the way the extended design does is 
> something a downstream consumer could also do on their side to ensure 
> anything that came in past the drop-off window wasn't already seen. So 
> not *too* painful; certainly a vast improvement over the status quo.
> 
> As to TCM and Accord: absolutely agree. I'd love to see a world where we 
> Accord everything and fire off CDC to subscribers from a coordinator 
> bypassing all this LSM-bastardized post-processing for CDC for instance. 
> That said, this is a functionality users needed back in... 2016? When we 
> first added CDC. So I think it's worth it to move on it now while 
> retaining architectural options to move to updated metadata and 
> transactions as they mature (obviously we'll lean on TCM since it's in 
> 5.0 / trunk right now; more applies to the accord bit).
> 
> On Mon, Sep 30, 2024, at 3:20 AM, Benedict wrote:
>> 
>> Yes, with accord it should be fairly easy to have reliable no-dupe log 
>> streaming without an elected leader. Given the broad set of use cases, I 
>> can imagine supporting some more native topic subscription API in C* 
>> rather than requiring Kafka, so perhaps any integration of Kafka with 
>> the sidecar can be considered a separate parallel effort, that might 
>> eventually implement itself with this C* feature whenever it 
>> materialises?
>> 
>> 
>>> On 30 Sep 2024, at 03:42, Jeff Jirsa  wrote:
>>> 
>>> 
>>> Transactional metadata and Accord should make it MUCH easier to do 
>>> duplication avoiding CDC (and I was going to note that someone should 
>>> ensu

Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

[João Reis]

+1 (nb)

Bret McGuire  escreveu (quinta, 26/09/2024 à(s)
21:46):

>Greetings all!
>
>I’m proposing the Cassandra Gocql Driver 1.7.0 for release.
>
> sha1: 953e0df999cabb3f5eef714df9921c00e9f632c2
> git: https://github.com/apache/cassandra-gocql-driver/tree/v1.7.0-rc1
>
>The Source release is available here:
>
> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-gocql-driver/1.7.0/
>
>This is the first release of the Gocql Driver since its donation.
> Developers will include this driver in their projects by specifying a
> commit hash or tag in go.mod rather than via the inclusion of binary
> artifacts so we’ve avoided the creation of binary artifacts completely.  To
> avoid any premature access to this release before the vote is complete
> we’ve temporarily used the tag “v1.7.0-rc1” to clearly indicate that this
> tag points to a release candidate.  Once the vote passes this tag will be
> updated to “v1.7.0”.
>
>The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
>
>Thanks all!
>
>   - Bret -
>

Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

[Bret McGuire]

   Vote passes with six +1 votes (3 binding).

   Thanks all!

  - Bret -

On Mon, Sep 30, 2024 at 10:19 AM João Reis  wrote:

> +1 (nb)
>
> Bret McGuire  escreveu (quinta, 26/09/2024 à(s)
> 21:46):
>
>>Greetings all!
>>
>>I’m proposing the Cassandra Gocql Driver 1.7.0 for release.
>>
>> sha1: 953e0df999cabb3f5eef714df9921c00e9f632c2
>> git: https://github.com/apache/cassandra-gocql-driver/tree/v1.7.0-rc1
>>
>>The Source release is available here:
>>
>> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-gocql-driver/1.7.0/
>>
>>This is the first release of the Gocql Driver since its donation.
>> Developers will include this driver in their projects by specifying a
>> commit hash or tag in go.mod rather than via the inclusion of binary
>> artifacts so we’ve avoided the creation of binary artifacts completely.  To
>> avoid any premature access to this release before the vote is complete
>> we’ve temporarily used the tag “v1.7.0-rc1” to clearly indicate that this
>> tag points to a release candidate.  Once the vote passes this tag will be
>> updated to “v1.7.0”.
>>
>>The vote will be open for 72 hours (longer if needed). Everyone who
>> has tested the build is invited to vote. Votes by PMC members are
>> considered binding. A vote passes if there are at least three binding +1s
>> and no -1's.
>>
>>Thanks all!
>>
>>   - Bret -
>>
>

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Bernardo Botella]

Thanks everyone for the comments.

Patrick:
The proposal includes a “best effort” approach for deduplication (some details 
can be found on the Digest class comments on the PR here 
https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
 ). That alone won’t eliminate all the duplicates, but as Josh points out, it 
moves the line to something way easier to handle for consumers, and definitely 
on the direction we should aim towards. Accord is definitely something this 
contribution will benefit from, that will move that line even further.

Benedict:
If I understand it correctly, your concern is that Kafka is somewhat the 
hardcoded option for a CDC stream being published? The proposal introduces a 
concept of data sources and sinks 
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks)
 being kafka the first implemented data sink. That means that the actual Kafka 
output should (will) be something pluggable.



> On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:
> 
>> I don't see much on how this would be handled other than "left to the end 
>> user to figure out." 
> My immediate thought when I read that was "Yes. But it's moving where we draw 
> the line of 'left to the end user to figure out' much further than it was 
> before".
> 
> This should only be necessary in edge cases w/extended severe degraded 
> availability where you can't hit QUORUM w/this design. So we go from "De-dupe 
> literally everything o ye' user" to "de-dupe a small fraction of a % of the 
> time when things really go off the rails".
> 
> It still leaves the burden of processing potential duplicates downstream, so 
> some complexity burden on the users remains if they have no tolerance for 
> processing duplicate messages, however the underlying machine resource 
> utilization (from "dedupe everything" to "dedupe a small % of things") is 
> pretty massively shifted by this design change. That, and using the hash of 
> the mutation the way the extended design does is something a downstream 
> consumer could also do on their side to ensure anything that came in past the 
> drop-off window wasn't already seen. So not too painful; certainly a vast 
> improvement over the status quo.
> 
> As to TCM and Accord: absolutely agree. I'd love to see a world where we 
> Accord everything and fire off CDC to subscribers from a coordinator 
> bypassing all this LSM-bastardized post-processing for CDC for instance. That 
> said, this is a functionality users needed back in... 2016? When we first 
> added CDC. So I think it's worth it to move on it now while retaining 
> architectural options to move to updated metadata and transactions as they 
> mature (obviously we'll lean on TCM since it's in 5.0 / trunk right now; more 
> applies to the accord bit).
> 
> On Mon, Sep 30, 2024, at 3:20 AM, Benedict wrote:
>> 
>> Yes, with accord it should be fairly easy to have reliable no-dupe log 
>> streaming without an elected leader. Given the broad set of use cases, I can 
>> imagine supporting some more native topic subscription API in C* rather than 
>> requiring Kafka, so perhaps any integration of Kafka with the sidecar can be 
>> considered a separate parallel effort, that might eventually implement 
>> itself with this C* feature whenever it materialises?
>> 
>> 
>>> On 30 Sep 2024, at 03:42, Jeff Jirsa  wrote:
>>> 
>>> 
>>> Transactional metadata and Accord should make it MUCH easier to do 
>>> duplication avoiding CDC (and I was going to note that someone should 
>>> ensure that the interfaces exposed to the public are stable enough not to 
>>> change the published api once those exist)
>>> 
>>> 
>>> 
 On Sep 29, 2024, at 7:04 PM, Patrick McFadin  wrote:
 
 As I was reviewing this, it occurred to me that it was talking about 
 Sidecar like it was a thing but that CEP has been stalled for quite some 
 time:  
 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
 
 If work on this is being done, should we get this official and wrapped up?
 
 On to the proposal...
 
 This has been a topic on the project for over 10 years now. I've seen 
 multiple goes at making this work and the issue that always turns out to 
 torpedo the project is handing dupes. To the point that they go from a 
 generalized Kafka producer engine to something specific to a particular 
 use case. I don't see much on how this would be handled other than "left 
 to the end user to figure out." 
 
 There is also little mention of where the increased resource load would be 
 handled. 
 
 This has been discussed many times before, but is it time to introduce the 
 concept of an elected leader for a token range for this type of operation? 
 It would eliminate a ton of problems that need to managed wh

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Josh McKenzie]

I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking through 
"how complex should it be to rip out the chronicle format, insert some other 
well defined and well known, and handle log rolling ourselves". My preference 
(which I didn't indicate earlier) would be to have that done independently.

There a reason we can't move forward with CEP-12 w/out addressing the chronicle 
stuff? i.e.
> I would like to have this resolved because there is CEP-12 I plan to deliver 
> and I hit this and I do not want to base that work on something we might 
> eventually abandon.
Why would CEP-12 be heavily coupled with chronicle? I would assume you'd be 
able to make light use of the existing logging + log rolling, and then someone 
else could come along and easily rip out chronicle and the rolling and add in a 
different one with minimal code changes?

On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
> I don't understand why CEP-12 should be a vehicle for introducing changes 
> like that. That is something totally unrelated. I am not going to be the one 
> to implement anything beyond CEP-12 and I am not the one who is going to 
> replace it either so if we make it a hard requirement for CEP-12 then CEP-12 
> as it is will never be introduced. Just want to be clear about that.
> 
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  wrote:
>> >
>> > I'd strongly support either rolling the format change into the CEP-12 
>> > proposal or having another CEP for introducing protobuf, spark, etc - some 
>> > kind of more broadly adopted format, and removing chronicle from our stack.
>> 
>> +1, I too would strongly support an open format and removing chronicle.
>> 
>> Kind Regards,
>> Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Štefan Miklošovič]

 "how complex should it be to rip out the chronicle format, insert some
other well defined and well known, and handle log rolling ourselves".

I think that it will be actually easier to do after CEP-12 is in because as
I mentioned it does housekeeping of what is there rigth now and refactors
it a little bit so throwing away the guts of it should be isolated only to
actual BinLog class and stuff around that which is built on top of
Chronicle.

"There a reason we can't move forward with CEP-12 w/out addressing the
chronicle stuff? i.e."

I think there is not any.

"Why would CEP-12 be heavily coupled with chronicle?" - it would not be
heavier from what is already there for audit of fql. Chronicle here
basically acts as a sink.

Actually, that patch also makes the implementations of (diagnostic too)
loggers pluggable  (via coding against an interface and putting that on the
class path) so people might already write it to whatever they want - even
to something protobuf-like. If they do not want to use Chronicle as a sink,
by implementing their own logger, they could just put it wherever they
want. I think that I forgot to mention this aspect. So the whole solution
we have is already not hard-wired to Chronicle necessarily. It is just
something we provide out of the box.

On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie  wrote:

> I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking
> through "how complex should it be to rip out the chronicle format, insert
> some other well defined and well known, and handle log rolling ourselves".
> My preference (which I didn't indicate earlier) would be to have that done
> independently.
>
> There a reason we can't move forward with CEP-12 w/out addressing the
> chronicle stuff? i.e.
>
> I would like to have this resolved because there is CEP-12 I plan to
> deliver and I hit this and I do not want to base that work on something we
> might eventually abandon.
>
> Why would CEP-12 be heavily coupled with chronicle? I would assume you'd
> be able to make light use of the existing logging + log rolling, and then
> someone else could come along and easily rip out chronicle and the rolling
> and add in a different one with minimal code changes?
>
> On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
>
> I don't understand why CEP-12 should be a vehicle for introducing changes
> like that. That is something totally unrelated. I am not going to be the
> one to implement anything beyond CEP-12 and I am not the one who is going
> to replace it either so if we make it a hard requirement for CEP-12 then
> CEP-12 as it is will never be introduced. Just want to be clear about that.
>
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>
> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie 
> wrote:
> >
> > I'd strongly support either rolling the format change into the CEP-12
> proposal or having another CEP for introducing protobuf, spark, etc - some
> kind of more broadly adopted format, and removing chronicle from our stack.
>
> +1, I too would strongly support an open format and removing chronicle.
>
> Kind Regards,
> Brandon
>
>
>

Re: CEP-15: Accord status

[Josh McKenzie]

> Today I learned… I had no clue we had markdown files in src/java…
Discoverability issues in our codebase?

Well I never.

;)

On Sat, Sep 28, 2024, at 10:39 PM, David Capwell wrote:
> Today I learned… I had no clue we had markdown files in src/java…
> 
> $ find src/ -name '*.md'
> src//java/org/apache/cassandra/io/sstable/SSTable_API.md
> src//java/org/apache/cassandra/io/sstable/format/bti/BtiFormat.md
> src//java/org/apache/cassandra/utils/bytecomparable/ByteComparable.md
> src//java/org/apache/cassandra/tcm/TCM_implementation.md
> src//java/org/apache/cassandra/tcm/TransactionalClusterMetadata.md
> src//java/org/apache/cassandra/db/memtable/Memtable_API.md
> src//java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.md
> src//java/org/apache/cassandra/db/tries/InMemoryTrie.md
> src//java/org/apache/cassandra/db/tries/Trie.md
> src//java/org/apache/cassandra/index/sai/README.md
> src//java/org/apache/cassandra/service/paxos/Paxos.md
> 
> 
> 
> We don’t have one at the moment but it would be good to get that in.  At a 
> high level there are a few key classes
> 
> 1) org.apache.cassandra.cql3.statements.TransactionStatement - this class 
> handles BEGIN TRANSACTION in CQL
> 2) org.apache.cassandra.service.consensus.TransactionalMode - this is a table 
> property and dictates what is allowed for the table.  If off accord 
> transactions are not allowed, if “full” normal read/write get migrated to 
> Accord (and you can still use BEGIN TRANSACTION)
> 3) org.apache.cassandra.service.accord.AccordService - the global static 
> instance that lets Cassandra call Accord stuff
> 
>> On Sep 27, 2024, at 7:20 AM, Paulo Motta  wrote:
>> 
>> Thanks all for the work on this epic!
>> 
>> Is there an implementation summary guide similar to guide_8099.md [1] that 
>> can help reviewers not involved with the effort navigate through the code ? 
>> It would be great to have it if this is not already available or being 
>> planned. There's a similar one though much smaller in scope for memtable API 
>> on [2].
>> 
>> [1] - 
>> https://github.com/apache/cassandra/blob/cassandra-3.0.0-rc2/guide_8099.md 
>> [2] - 
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
>> 
>> On Fri, Sep 27, 2024 at 8:09 AM Benedict Elliott Smith  
>> wrote:
>>> If you exclude test changes, there’s < 50k added and ~2k removed. This 
>>> works out to ~7% of the scale of 8099 for lines modified, if this is the 
>>> benchmark for disruption.
>>> 
>>> Altogether, this is a very small patch from the perspective of the existing 
>>> codebase. Probably doesn’t even come close to the top 10.
>>> 
>>> Conversely, for new standalone features, this is likely the most complex 
>>> thing we have ever merged to the project. But, it is off by default, and 
>>> the risk to deployments therefore is very minimal. 
>>> 
>>> Regarding how parties can engage, I think if we’re honest history shows 
>>> that engagement will be minimal. There have after all been several touch 
>>> points, and none have materialised into really significant engagement. This 
>>> is just the reality of everyone having their own pressures - at the end of 
>>> the day, changes happen and the community adapts. But, we are here to 
>>> answer any questions - as we have been throughout the development of the 
>>> work in the open.
>>> 
>>> 
>>> 
 On 20 Sep 2024, at 22:08, Josh McKenzie  wrote:
 
> This presents an opportune moment for those interested to review the code.
> ...
> +88,341 −7,341
> 1003 Files changed
 
 O.o 
 This is... **very large**. If we use CASSANDRA-8099 as our "banana for 
 scale":
> 645 files changed, 49381 insertions(+), 42227 deletions(-)
 
 To be clear - I don't think we collectively should be worried about 
 disruption from this patch since:
  1. Each commit (or the vast majority?) has already been reviewed by >= 1 
 other committer
  2. 7.3k deletions is a lot less than 42k
  3. We now have fuzzing, property based testing, and the simulator
  4. Most of this code is additive
 How would you recommend interested parties engage with reviewing this 
 behemoth? Or perhaps subsections of it or key areas to familiarize 
 themselves with the structure?
 
 On Fri, Sep 20, 2024, at 12:17 PM, David Capwell wrote:
> Recently, we rebased against the trunk branch, ensuring that the accord 
> branch is now in sync with the latest trunk version. This presents an 
> opportune moment for those interested to review the code.
> 
> We have a pending pull request 
> (_https://github.com/apache/cassandra/pull/3552_) that we do not intend 
> to merge.
> 
> Our current focus is on addressing several bug fixes and ensuring the 
> safety of topology changes (as evidenced by the number of issues filed 
> against the trunk). Once we wrap up bug fixes and safety features, we 
> will likely

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Josh McKenzie]

> thinking more about stuff like protobuf and while I do see benefits of that, 
> honestly, it just does not matter too much if it is done like that or not.
I disagree; I think having a language and environment agnostic file format 
matters a great deal. Unless we're talking specifically about the FQL case in 
which case I totally agree. :)

Part of what makes open source projects successful is their ability to 
modularly interop with other projects, often times in ways we don't predict; us 
using the Chronicle format for our binary files is choosing a format that's 
esoteric with fairly antagonistic licensing compared to something like 
protobufs.

I'd strongly support either rolling the format change into the CEP-12 proposal 
or having another CEP for introducing protobuf, spark, etc - some kind of more 
broadly adopted format, and removing chronicle from our stack.

To Jon's point: having bespoke formats with all the code for it housed in 
cassandra-all rather than in a modular support library really ties our hands on 
a lot of fronts; updating the core DB in terms of serialization formats, of JDK 
versions, etc all potentially become client-impacting exercises.


On Mon, Sep 30, 2024, at 5:12 AM, Štefan Miklošovič wrote:
> That is all OK to mention. So as I read it correctly, Jon, myself, Andrew, 
> David - we all would like to see some cross-language events ingestion. Other 
> people do not seem to consider it important enough (just correct me if I am 
> wrong). I do not mind, I am 50:50, not an absolute must but sure, let's add 
> that to the wish list.
> 
> I believe that unless it is an absolute show-stopper (which it is not) it is 
> not necessary. It is something to keep in mind upon next refactorization. I 
> am personally not affected by this. Whoever will need that, they will find a 
> way to make it happen.
> 
> I would really appreciate it if we found consensus among these points I wrote 
> down. From my point of view, the most ideal outcome would be to make CEP-12 
> happen which cleans it all up and makes it more robust.
> 
> My perception is that we have always found the most practical solution and if 
> it brings value to a user (being able to inspect diagnostic events offline 
> for further inspection) we should not avoid delivering that. Something 
> similar happened in e.g. Password validator / generator (CEP-24) where the 
> most ideal solution was to base it on transactional configuration but even 
> though we are not there yet, that did not stop us from delivering it because 
> for some entities in this space it brought value anyway. 
> 
> It may seem as if I am invested in delivering it because I spent some time on 
> that already - that is not the case - I am OK to drop diagnostic events 
> persistence if you insist on that but honestly I just do not see any 
> compelling argument to do so. The library we are using that for (Chronicle 
> Queues) is already there, it is just a functionality addition and as I said, 
> if somebody comes and delivers a solution which would replace it, I do not 
> see it to be problematic. It would be replaced as anything else (FQL, Audit 
> ...). 
> 
> On Sun, Sep 29, 2024 at 9:06 PM Jon Haddad  wrote:
>> Strong +1 to the file format issue, and if we're building a wish list - it 
>> would be great if we could read the file format without pulling in 
>> cassandra-all.  Long term, I'd love to see this for SSTables & Commit logs 
>> as well.
>> 
>> I've long been a fan of Gradle subprojects because it makes this kind of 
>> thing fairly easy. 
>> 
>> Jon
>> 
>> 
>> 
>> On Sun, Sep 29, 2024 at 11:46 AM Andrew Weaver  
>> wrote:
>>> I'm late to the discussion here, but I want to add my experience from 
>>> dealing with audit logs specifically. 
>>> 
>>> Chronicle has some advantages (binary, compact) but it has a serious 
>>> disadvantage from a consumption standpoint. It's not a well-supported file 
>>> format. Audit logs are something that I think most operators are interested 
>>> in archiving for compliance purposes and analyzing offline for any number 
>>> of reasons and an oddball file format is an unnecessary hurdle for the 
>>> audit logs use-case. 
>>> 
>>> I would welcome support for an existing format that is compact, 
>>> high-performance and compatible with common tools (Spark, etc.).
>>> 
>>> On Sun, Sep 29, 2024, 10:11 AM Štefan Miklošovič  
>>> wrote:
 Thank you all for your answers and opinions. I would like to have some 
 kind of a resolution here in order to move forward, especially with 
 relation to CEP-12 I mentioned earlier. (1).
 
 I think we have these options:
 
 1) Do nothing and wait until this gets back to us, probably in a more 
 serious way (we find a bug and we will not be able to update it because it 
 would be on "ea" or new features will be available only in newer versions).
 
 2) Fork it and continue to maintain it - I do not think this is realistic, 
 nobody is going to t

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Jon Haddad]

I think it depends on what lens you're looking at the sidecar through.

If you're actively working on it, and pulling it into your own infra,
sure.  It's a thing.

If you're an outsider?  I have a hard time seeing it.

- No documentation as to what it does
- No releases
- No build instructions
- Trying to build using standard gradle commands fails [1]
- Included configs don't work out of the box. [2][3]
- CEP-1 looks abandonded
- The primary reason right now to use it looks to be analytics library,
which doesn't work for most teams due to lack of vnode support [4]

I think if you were to take a poll of 100 users outside this ML, I'd bet
almost every one would agree the sidecar isn't a thing yet, and that's
probably more important than if it's actually getting worked on.  I think
it has quite a ways to go before it looks to be more than an idea that's
incubating.

[1] https://issues.apache.org/jira/browse/CASSANDRASC-120
[2 https://issues.apache.org/jira/browse/CASSANDRASC-121
[3] https://issues.apache.org/jira/browse/CASSANDRASC-122
[4] https://issues.apache.org/jira/browse/CASSANDRA-19594


On Mon, Sep 30, 2024 at 11:14 AM Josh McKenzie  wrote:

> The CEP for the sidecar has stalled. The sidecar itself is very much alive
> and a thing.
>
> CEP != artifact.
>
> We should definitely clean that up though.
>
> On Mon, Sep 30, 2024, at 10:59 AM, Dinesh Joshi wrote:
>
> Patrick, could you please elaborate? The Sidecar has been a thing for a
> while now.
>
> On Mon, Sep 30, 2024 at 7:51 AM Patrick McFadin 
> wrote:
>
> I made the mistake of asking two things in one email.
>
> First thing I asked. Sidecar? Stalled CEP so why is this being talked
> about like this is a thing?
>
> On Mon, Sep 30, 2024 at 7:21 AM Benedict  wrote:
>
>
> Sorry Bernardo, you may have misunderstood me. I don’t have any concerns,
> I was suggesting a possible future scenario where CDC for Kafka via sidecar
> is changed to use a hypothetical future topic subscription service provided
> by C*. It was meant to show that this CEP may be easily decoupled from any
> future evolution in this area.
>
>
> On 30 Sep 2024, at 14:58, Bernardo Botella 
> wrote:
>
> Thanks everyone for the comments.
>
>
> Patrick:
> The proposal includes a “best effort” approach for deduplication (some
> details can be found on the Digest class comments on the PR here
> https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
>  ).
> That alone won’t eliminate all the duplicates, but as Josh points out, it
> moves the line to something way easier to handle for consumers, and
> definitely on the direction we should aim towards. Accord is definitely
> something this contribution will benefit from, that will move that line
> even further.
>
> Benedict:
> If I understand it correctly, your concern is that Kafka is somewhat the
> hardcoded option for a CDC stream being published? The proposal introduces
> a concept of data sources and sinks (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks)
> being kafka the first implemented data sink. That means that the actual
> Kafka output should (will) be something pluggable.
>
>
>
> On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:
>
> I don't see much on how this would be handled other than "left to the end
> user to figure out."
>
> My immediate thought when I read that was "Yes. But it's moving where we
> draw the line of 'left to the end user to figure out' *much further* than
> it was before".
>
> This should only be necessary in edge cases w/extended severe degraded
> availability where you can't hit QUORUM w/this design. So we go from
> "De-dupe literally everything o ye' user" to "de-dupe a small fraction of a
> % of the time when things really go off the rails".
>
> It still leaves the burden of processing potential duplicates downstream,
> so some *complexity* burden on the users remains if they have no
> tolerance for processing duplicate messages, however the underlying machine
> resource utilization (from "dedupe everything" to "dedupe a small % of
> things") is pretty massively shifted by this design change. That, and using
> the hash of the mutation the way the extended design does is something a
> downstream consumer could also do on their side to ensure anything that
> came in past the drop-off window wasn't already seen. So not *too* painful;
> certainly a vast improvement over the status quo.
>
> As to TCM and Accord: absolutely agree. I'd love to see a world where we
> Accord everything and fire off CDC to subscribers from a coordinator
> bypassing all this LSM-bastardized post-processing for CDC for instance.
> That said, this is a functionality users needed back in... 2016? When we
> first added CDC. So I think it's worth it to move on it now while retaining
> architectural options to move to updated metadata and transactions as they
>

Re: Status of CEP-1

[Patrick McFadin]

There are two easy choices.

1 - Re-furbish CEP-1 and start a [DISCUSS] thread
2 - Close out CEP-1 and Propose something fresh and start a [DISCUSS]
Thread on that.

Do you think there is enough in CEP-1 to keep moving with or is it
completely wrong?

Patrick

On Mon, Sep 30, 2024 at 4:53 PM Francisco Guerrero 
wrote:

> Hi folks,
>
> I feel I need to update the status of CEP-1 as it currently stands.
> For context, the Cassandra Sidecar project has had a steady flow of
> contributions in the past couple of years. And there is a steady stream
> of upcoming contributions, i.e live migration (CEP-40), CDC (CEP-44),
> and many others. However, I believe we need to address one issue
> with CEP-1; and that is its scope.
>
> The scope of CEP-1 is too broad, and I would like to propose either
> closing on CEP-1 or rescoping it. We have a Sidecar now, it's part of
> the foundation, and AFAIK we've pretty much satisfied the 2 goals of
> CEP-1 which are listed as "extensible and passes the curl test" and
> "provides basic but essential and useful functionality".
>
> CEP-1 was discussed and consensus was achieved in 2018 after
> a lot of discussion[4]. CEP-1 contributed to the foundation of the CEP
> process. Several JIRAs have been opened and active contribution is
> happening in the subproject.
>
> We are getting close to proposing the first release of Sidecar, pending
> some trivial fixes needed in the configuration and build
> processes[1][2][3];
> as well as CASSANDRASC-141[5] which will bring authn/authz into Sidecar.
> Once
> we close on CASSANDRASC-141, Sidecar will be ready for the 1.0 release.
> Any new major feature to Sidecar would go through the regular CEP process.
>
> Cassandra’s Sidecar usage is not restricted to the Analytics library,
> however
> it does support this use case at the moment. I will not touch on vnode
> support in Cassandra Analytics as it deserves its own separate discussion.
>
> We're excited to invite you to a talk on Cassandra Sidecar at the Community
> Over Code next week. Join us as we explore the current features and share
> what’s on the horizon for Sidecar.
>
> Looking forward to hearing your thoughts on this proposal.
>
> Best,
> ⁃ Francisco
>
> [1] https://issues.apache.org/jira/browse/CASSANDRASC-120
> [2] https://issues.apache.org/jira/browse/CASSANDRASC-121
> [3] https://issues.apache.org/jira/browse/CASSANDRASC-122
> [4] https://lists.apache.org/thread/xyg8n5hkt7xrfqv48k91tx1jwp0pvcpw
> [5] https://issues.apache.org/jira/browse/CASSANDRASC-141
>
>
>
>

Re: Status of CEP-1

[Dinesh Joshi]

CEP-1 is still completely relevant and we could send an update but as it
stands right now we’ve made a ton of progress and would like to focus on
getting to a release so it’s real for the community.

On Mon, Sep 30, 2024 at 5:31 PM Patrick McFadin  wrote:

> There are two easy choices.
>
> 1 - Re-furbish CEP-1 and start a [DISCUSS] thread
> 2 - Close out CEP-1 and Propose something fresh and start a [DISCUSS]
> Thread on that.
>
> Do you think there is enough in CEP-1 to keep moving with or is it
> completely wrong?
>
> Patrick
>
> On Mon, Sep 30, 2024 at 4:53 PM Francisco Guerrero 
> wrote:
>
>> Hi folks,
>>
>> I feel I need to update the status of CEP-1 as it currently stands.
>> For context, the Cassandra Sidecar project has had a steady flow of
>> contributions in the past couple of years. And there is a steady stream
>> of upcoming contributions, i.e live migration (CEP-40), CDC (CEP-44),
>> and many others. However, I believe we need to address one issue
>> with CEP-1; and that is its scope.
>>
>> The scope of CEP-1 is too broad, and I would like to propose either
>> closing on CEP-1 or rescoping it. We have a Sidecar now, it's part of
>> the foundation, and AFAIK we've pretty much satisfied the 2 goals of
>> CEP-1 which are listed as "extensible and passes the curl test" and
>> "provides basic but essential and useful functionality".
>>
>> CEP-1 was discussed and consensus was achieved in 2018 after
>> a lot of discussion[4]. CEP-1 contributed to the foundation of the CEP
>> process. Several JIRAs have been opened and active contribution is
>> happening in the subproject.
>>
>> We are getting close to proposing the first release of Sidecar, pending
>> some trivial fixes needed in the configuration and build
>> processes[1][2][3];
>> as well as CASSANDRASC-141[5] which will bring authn/authz into Sidecar.
>> Once
>> we close on CASSANDRASC-141, Sidecar will be ready for the 1.0 release.
>> Any new major feature to Sidecar would go through the regular CEP process.
>>
>> Cassandra’s Sidecar usage is not restricted to the Analytics library,
>> however
>> it does support this use case at the moment. I will not touch on vnode
>> support in Cassandra Analytics as it deserves its own separate discussion.
>>
>> We're excited to invite you to a talk on Cassandra Sidecar at the
>> Community
>> Over Code next week. Join us as we explore the current features and share
>> what’s on the horizon for Sidecar.
>>
>> Looking forward to hearing your thoughts on this proposal.
>>
>> Best,
>> ⁃ Francisco
>>
>> [1] https://issues.apache.org/jira/browse/CASSANDRASC-120
>> [2] https://issues.apache.org/jira/browse/CASSANDRASC-121
>> [3] https://issues.apache.org/jira/browse/CASSANDRASC-122
>> [4] https://lists.apache.org/thread/xyg8n5hkt7xrfqv48k91tx1jwp0pvcpw
>> [5] https://issues.apache.org/jira/browse/CASSANDRASC-141
>>
>>
>>
>>

Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

[Brandon Williams]

+1

Kind Regards,
Brandon

On Thu, Sep 26, 2024 at 3:46 PM Bret McGuire  wrote:
>
>Greetings all!
>
>I’m proposing the Cassandra Gocql Driver 1.7.0 for release.
>
> sha1: 953e0df999cabb3f5eef714df9921c00e9f632c2
> git: https://github.com/apache/cassandra-gocql-driver/tree/v1.7.0-rc1
>
>The Source release is available here:
> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-gocql-driver/1.7.0/
>
>This is the first release of the Gocql Driver since its donation.  
> Developers will include this driver in their projects by specifying a commit 
> hash or tag in go.mod rather than via the inclusion of binary artifacts so 
> we’ve avoided the creation of binary artifacts completely.  To avoid any 
> premature access to this release before the vote is complete we’ve 
> temporarily used the tag “v1.7.0-rc1” to clearly indicate that this tag 
> points to a release candidate.  Once the vote passes this tag will be updated 
> to “v1.7.0”.
>
>The vote will be open for 72 hours (longer if needed). Everyone who has 
> tested the build is invited to vote. Votes by PMC members are considered 
> binding. A vote passes if there are at least three binding +1s and no -1's.
>
>Thanks all!
>
>   - Bret -

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Brandon Williams]

On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  wrote:
>
> I'd strongly support either rolling the format change into the CEP-12 
> proposal or having another CEP for introducing protobuf, spark, etc - some 
> kind of more broadly adopted format, and removing chronicle from our stack.

+1, I too would strongly support an open format and removing chronicle.

Kind Regards,
Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Štefan Miklošovič]

I don't understand why CEP-12 should be a vehicle for introducing changes
like that. That is something totally unrelated. I am not going to be the
one to implement anything beyond CEP-12 and I am not the one who is going
to replace it either so if we make it a hard requirement for CEP-12 then
CEP-12 as it is will never be introduced. Just want to be clear about that.

On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:

> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie 
> wrote:
> >
> > I'd strongly support either rolling the format change into the CEP-12
> proposal or having another CEP for introducing protobuf, spark, etc - some
> kind of more broadly adopted format, and removing chronicle from our stack.
>
> +1, I too would strongly support an open format and removing chronicle.
>
> Kind Regards,
> Brandon
>

Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

[Josh McKenzie]

+1. SHAs and contents lgtm.

On Mon, Sep 30, 2024, at 5:57 AM, Dmytro Tsapko wrote:
> Hello, LGTM.
> 
> On Thu, Sep 26, 2024 at 11:46 PM Bret McGuire  wrote:
>>Greetings all!
>> 
>>I’m proposing the Cassandra Gocql Driver 1.7.0 for release.
>> 
>> sha1: 953e0df999cabb3f5eef714df9921c00e9f632c2
>> git: https://github.com/apache/cassandra-gocql-driver/tree/v1.7.0-rc1
>> 
>>The Source release is available here:
>> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-gocql-driver/1.7.0/
>> 
>>This is the first release of the Gocql Driver since its donation.  
>> Developers will include this driver in their projects by specifying a commit 
>> hash or tag in go.mod rather than via the inclusion of binary artifacts so 
>> we’ve avoided the creation of binary artifacts completely.  To avoid any 
>> premature access to this release before the vote is complete we’ve 
>> temporarily used the tag “v1.7.0-rc1” to clearly indicate that this tag 
>> points to a release candidate.  Once the vote passes this tag will be 
>> updated to “v1.7.0”.
>> 
>>The vote will be open for 72 hours (longer if needed). Everyone who has 
>> tested the build is invited to vote. Votes by PMC members are considered 
>> binding. A vote passes if there are at least three binding +1s and no -1's.
>> 
>>Thanks all!
>> 
>>   - Bret -

[RESULT][VOTE] Release Apache Cassandra 5.0.1

[Mick Semb Wever]

.

Proposing the test build of Cassandra 5.0.1 for release.
>
> sha1: c206e4509003ac4cd99147d821bd4b5d23bdf5e8
> Git: https://github.com/apache/cassandra/tree/5.0.1-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1348/org/apache/cassandra/cassandra-all/5.0.1/
>
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/5.0.1/
>
> ==
> This release introduces safeguards and observability into possible data
> loss scenarios when nodes have a  divergent view of the cluster. This
> happens around edge-cases on unsafe bootstrapping, decommissions, or when a
> node has a corrupted topology. Two configuration options have been added:
> `log_out_of_token_range_requests` and
>  `reject_out_of_token_range_requests`, both enabled by default. The former
> will make nodes log requests they receive that don't belong in their
> current or pending token ranges. The latter will reject those requests,
> which prevents any eventual data loss that can occur but may also incur
> small windows of degraded availability during range movements. See
> CASSANDRA-13704 for further details.
> ==
>
> The vote will be open for 96 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
>


Vote passes with four +1s, three binding.

Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

[Štefan Miklošovič]

+1

On Thu, Sep 26, 2024 at 10:49 PM Bret McGuire 
wrote:

>Greetings all!
>
>I’m proposing the Cassandra Gocql Driver 1.7.0 for release.
>
> sha1: 953e0df999cabb3f5eef714df9921c00e9f632c2
> git: https://github.com/apache/cassandra-gocql-driver/tree/v1.7.0-rc1
>
>The Source release is available here:
>
> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-gocql-driver/1.7.0/
>
>This is the first release of the Gocql Driver since its donation.
> Developers will include this driver in their projects by specifying a
> commit hash or tag in go.mod rather than via the inclusion of binary
> artifacts so we’ve avoided the creation of binary artifacts completely.  To
> avoid any premature access to this release before the vote is complete
> we’ve temporarily used the tag “v1.7.0-rc1” to clearly indicate that this
> tag points to a release candidate.  Once the vote passes this tag will be
> updated to “v1.7.0”.
>
>The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
>
>Thanks all!
>
>   - Bret -
>

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Benedict]

Sorry Bernardo, you may have misunderstood me. I don’t have any concerns, I was suggesting a possible future scenario where CDC for Kafka via sidecar is changed to use a hypothetical future topic subscription service provided by C*. It was meant to show that this CEP may be easily decoupled from any future evolution in this area. On 30 Sep 2024, at 14:58, Bernardo Botella  wrote:Thanks everyone for the comments.Patrick:The proposal includes a “best effort” approach for deduplication (some details can be found on the Digest class comments on the PR here https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193 ). That alone won’t eliminate all the duplicates, but as Josh points out, it moves the line to something way easier to handle for consumers, and definitely on the direction we should aim towards. Accord is definitely something this contribution will benefit from, that will move that line even further.Benedict:If I understand it correctly, your concern is that Kafka is somewhat the hardcoded option for a CDC stream being published? The proposal introduces a concept of data sources and sinks (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks) being kafka the first implemented data sink. That means that the actual Kafka output should (will) be something pluggable.On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:I don't see much on how this would be handled other than "left to the end user to figure out." My immediate thought when I read that was "Yes. But it's moving where we draw the line of 'left to the end user to figure out' much further than it was before".This should only be necessary in edge cases w/extended severe degraded availability where you can't hit QUORUM w/this design. So we go from "De-dupe literally everything o ye' user" to "de-dupe a small fraction of a % of the time when things really go off the rails".It still leaves the burden of processing potential duplicates downstream, so some complexity burden on the users remains if they have no tolerance for processing duplicate messages, however the underlying machine resource utilization (from "dedupe everything" to "dedupe a small % of things") is pretty massively shifted by this design change. That, and using the hash of the mutation the way the extended design does is something a downstream consumer could also do on their side to ensure anything that came in past the drop-off window wasn't already seen. So not too painful; certainly a vast improvement over the status quo.As to TCM and Accord: absolutely agree. I'd love to see a world where we Accord everything and fire off CDC to subscribers from a coordinator bypassing all this LSM-bastardized post-processing for CDC for instance. That said, this is a functionality users needed back in... 2016? When we first added CDC. So I think it's worth it to move on it now while retaining architectural options to move to updated metadata and transactions as they mature (obviously we'll lean on TCM since it's in 5.0 / trunk right now; more applies to the accord bit).On Mon, Sep 30, 2024, at 3:20 AM, Benedict wrote:Yes, with accord it should be fairly easy to have reliable no-dupe log streaming without an elected leader. Given the broad set of use cases, I can imagine supporting some more native topic subscription API in C* rather than requiring Kafka, so perhaps any integration of Kafka with the sidecar can be considered a separate parallel effort, that might eventually implement itself with this C* feature whenever it materialises?On 30 Sep 2024, at 03:42, Jeff Jirsa  wrote:Transactional metadata and Accord should make it MUCH easier to do duplication avoiding CDC (and I was going to note that someone should ensure that the interfaces exposed to the public are stable enough not to change the published api once those exist)On Sep 29, 2024, at 7:04 PM, Patrick McFadin  wrote:As I was reviewing this, it occurred to me that it was talking about Sidecar like it was a thing but that CEP has been stalled for quite some time:  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224If work on this is being done, should we get this official and wrapped up?On to the proposal...This has been a topic on the project for over 10 years now. I've seen multiple goes at making this work and the issue that always turns out to torpedo the project is handing dupes. To the point that they go from a generalized Kafka producer engine to something specific to a particular use case. I don't see much on how this would be handled other than "left to the end user to figure out." There is also little mention of where the increased resource load would be handled. This has been discussed many times before, but is it time to introduce the concept of an elected leader for a token range for this type of operation? It would eliminate a ton of 

Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

[Rolo, Carlos via dev]

My vote doesn't count, but this would be a +1 from me.

From: Brandon Williams 
Sent: 30 September 2024 14:06
To: dev@cassandra.apache.org 
Subject: Re: [VOTE] Release Apache Cassandra Gocql Driver 1.7.0

EXTERNAL EMAIL - USE CAUTION when clicking links or attachments




+1

Kind Regards,
Brandon

On Thu, Sep 26, 2024 at 3:46 PM Bret McGuire  wrote:
>
>Greetings all!
>
>I’m proposing the Cassandra Gocql Driver 1.7.0 for release.
>
> sha1: 953e0df999cabb3f5eef714df9921c00e9f632c2
> git: 
> https://github.com/apache/cassandra-gocql-driver/tree/v1.7.0-rc1
>
>The Source release is available here:
> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-gocql-driver/1.7.0/
>
>This is the first release of the Gocql Driver since its donation.  
> Developers will include this driver in their projects by specifying a commit 
> hash or tag in go.mod rather than via the inclusion of binary artifacts so 
> we’ve avoided the creation of binary artifacts completely.  To avoid any 
> premature access to this release before the vote is complete we’ve 
> temporarily used the tag “v1.7.0-rc1” to clearly indicate that this tag 
> points to a release candidate.  Once the vote passes this tag will be updated 
> to “v1.7.0”.
>
>The vote will be open for 72 hours (longer if needed). Everyone who has 
> tested the build is invited to vote. Votes by PMC members are considered 
> binding. A vote passes if there are at least three binding +1s and no -1's.
>
>Thanks all!
>
>   - Bret -

Re: CEP-32: Open-Telemetry integration

[Yuki Morishita]

Hi,

Since I have limited time working on the CEP-32, I'd appreciate the
collaboration to make this CEP the reality.

Another thing I'm thinking of is to reduce its scope to only the
OpenTelemetry configuration and the way to work only with OpenTelemetry
Tracing.

If it's possible to create sub CEPs, I will create the one for tracing,
metrics and logs. Otherwise, I can rewrite the current CEP-32 to only focus
on OpenTelemetry Tracing.
Or maybe scrap CEP-32 and create a new one for Tracing.


On Mon, Sep 23, 2024 at 11:47 AM Saranya Krishnakumar <
saran.krishna...@gmail.com> wrote:

> Hi Patrick,
>
> I am interested in working on this CEP collaborating with Yuki. I recently
> worked on adding metrics framework in Apache Cassandra Sidecar project.
>
> Best,
> Saranya Krishnakumar
>
> On Thu, Sep 19, 2024 at 10:57 AM Patrick McFadin 
> wrote:
>
>> Here's another stalled CEP. In this case, no discuss thread or Jira.
>>
>> Yuki (or anyone else) know the status of this CEP?
>>
>>
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-32%3A+%28DRAFT%29+OpenTelemetry+integration
>>
>> Patrick
>>
>

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Josh McKenzie]

> This is the type of hidden subproject that will get us into trouble with the 
> board/foundation.   I'm sure it's getting enough committer eyeballs, and some 
> PMC oversight, but maybe not enough.
I don't agree with the qualifier of it as being hidden. It's definitely lower 
traffic than the main project but there's movement on the JIRA here: link 
.

I assume the sidecar is going to take longer to reach a tipping point where 
more people start contributing to it until it has compelling features that'll 
incentivize folks running their own bespoke sidecars to migrate over.

Agree with all your points Jon; there's a lot to be done on it.

CEP-1 is pretty much abandoned yeah. I think it'd be reasonable to close it 
down and open up a new one w/active contributors + active shepherd and a much 
more limited scope.

On Mon, Sep 30, 2024, at 2:13 PM, Patrick McFadin wrote:
> I'm mentioning it because I was surprised and I feel like I generally have a 
> finger on the pulse of the project.
> 
> I would love to talk about it more and get more community support and 
> interest.
> 
> On Mon, Sep 30, 2024 at 11:01 AM Mick Semb Wever  wrote:
>> Agree with Jon, Josh and Patrick here.
>> 
>> This is the type of hidden subproject that will get us into trouble with the 
>> board/foundation.   I'm sure it's getting enough committer eyeballs, and 
>> some PMC oversight, but maybe not enough.  Addressing the more material 
>> points that Jon mentions is the best way to deal with that IMHO.
>> 
>> 
>> 
>> On Mon, 30 Sept 2024 at 20:37, Jon Haddad  wrote:
>>> I think it depends on what lens you're looking at the sidecar through.
>>> 
>>> If you're actively working on it, and pulling it into your own infra, sure. 
>>>  It's a thing. 
>>> 
>>> If you're an outsider?  I have a hard time seeing it.
>>> 
>>> - No documentation as to what it does
>>> - No releases
>>> - No build instructions
>>> - Trying to build using standard gradle commands fails [1]
>>> - Included configs don't work out of the box. [2][3]
>>> - CEP-1 looks abandonded
>>> - The primary reason right now to use it looks to be analytics library, 
>>> which doesn't work for most teams due to lack of vnode support [4]
>>> 
>>> I think if you were to take a poll of 100 users outside this ML, I'd bet 
>>> almost every one would agree the sidecar isn't a thing yet, and that's 
>>> probably more important than if it's actually getting worked on.  I think 
>>> it has quite a ways to go before it looks to be more than an idea that's 
>>> incubating.
>>> 
>>> [1] https://issues.apache.org/jira/browse/CASSANDRASC-120
>>> [2 https://issues.apache.org/jira/browse/CASSANDRASC-121
>>> [3] https://issues.apache.org/jira/browse/CASSANDRASC-122
>>> [4] https://issues.apache.org/jira/browse/CASSANDRA-19594
>>> 
>>> 
>>> On Mon, Sep 30, 2024 at 11:14 AM Josh McKenzie  wrote:
 __
 The CEP for the sidecar has stalled. The sidecar itself is very much alive 
 and a thing.
 
 CEP != artifact.
 
 We should definitely clean that up though.
 
 On Mon, Sep 30, 2024, at 10:59 AM, Dinesh Joshi wrote:
> Patrick, could you please elaborate? The Sidecar has been a thing for a 
> while now.
> 
> On Mon, Sep 30, 2024 at 7:51 AM Patrick McFadin  
> wrote:
>> I made the mistake of asking two things in one email. 
>> 
>> First thing I asked. Sidecar? Stalled CEP so why is this being talked 
>> about like this is a thing?
>> 
>> On Mon, Sep 30, 2024 at 7:21 AM Benedict  wrote:
>>> 
>>> Sorry Bernardo, you may have misunderstood me. I don’t have any 
>>> concerns, I was suggesting a possible future scenario where CDC for 
>>> Kafka via sidecar is changed to use a hypothetical future topic 
>>> subscription service provided by C*. It was meant to show that this CEP 
>>> may be easily decoupled from any future evolution in this area. 
>>> 
>>> 
 On 30 Sep 2024, at 14:58, Bernardo Botella 
  wrote:
 Thanks everyone for the comments.
 
 Patrick:
 The proposal includes a “best effort” approach for deduplication (some 
 details can be found on the Digest class comments on the PR here 
 https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
  ). That alone won’t eliminate all the duplicates, but as Josh points 
 out, it moves the line to something way easier to handle for 
 consumers, and definitely on the direction we should aim towards. 
 Accord is definitely something this contribution will benefit from, 
 that will move that line even further.
 
 Benedict:
 If I understand it correctly, your concern is that Kafka is somewhat 
 the hardcoded 

Status of CEP-1

[Francisco Guerrero]

Hi folks,

I feel I need to update the status of CEP-1 as it currently stands.
For context, the Cassandra Sidecar project has had a steady flow of
contributions in the past couple of years. And there is a steady stream
of upcoming contributions, i.e live migration (CEP-40), CDC (CEP-44),
and many others. However, I believe we need to address one issue
with CEP-1; and that is its scope.

The scope of CEP-1 is too broad, and I would like to propose either
closing on CEP-1 or rescoping it. We have a Sidecar now, it's part of
the foundation, and AFAIK we've pretty much satisfied the 2 goals of
CEP-1 which are listed as "extensible and passes the curl test" and
"provides basic but essential and useful functionality".

CEP-1 was discussed and consensus was achieved in 2018 after
a lot of discussion[4]. CEP-1 contributed to the foundation of the CEP
process. Several JIRAs have been opened and active contribution is
happening in the subproject.

We are getting close to proposing the first release of Sidecar, pending
some trivial fixes needed in the configuration and build processes[1][2][3];
as well as CASSANDRASC-141[5] which will bring authn/authz into Sidecar.
Once
we close on CASSANDRASC-141, Sidecar will be ready for the 1.0 release.
Any new major feature to Sidecar would go through the regular CEP process.

Cassandra’s Sidecar usage is not restricted to the Analytics library,
however
it does support this use case at the moment. I will not touch on vnode
support in Cassandra Analytics as it deserves its own separate discussion.

We're excited to invite you to a talk on Cassandra Sidecar at the Community
Over Code next week. Join us as we explore the current features and share
what’s on the horizon for Sidecar.

Looking forward to hearing your thoughts on this proposal.

Best,
⁃ Francisco

[1] https://issues.apache.org/jira/browse/CASSANDRASC-120
[2] https://issues.apache.org/jira/browse/CASSANDRASC-121
[3] https://issues.apache.org/jira/browse/CASSANDRASC-122
[4] https://lists.apache.org/thread/xyg8n5hkt7xrfqv48k91tx1jwp0pvcpw
[5] https://issues.apache.org/jira/browse/CASSANDRASC-141

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Štefan Miklošovič]

Okay, that works, so summary:

1) CEP-12 will be delivered WITHOUT interacting with Chronicle Queues,
something else will be used (most probably just a simple logger logging to
a text file, that is just enough for something like diagnostic logs which
are logged very sparsely)

2) We will NOT use Chronicle in the future for any new implementations /
solutions from now on.

3) We WILL be replacing loggers based on Chronicle Queues.

4) A new implementation replacing Chronicle Queues SHOULD take into account
a heterogeneous environment when it comes to consumers of the events
logged, being language-agnostic or supporting an open format which enables
that.

On Mon, Sep 30, 2024 at 5:07 PM Josh McKenzie  wrote:

> I say we do CEP-12 w/out using chronicle and have a follow up JIRA to
> replace chronicle w/something else.
>
> Seems like there's a reasonable consensus around at least that subset of
> things given the discussion here. As to what form that something else
> takes, well, that's a topic for another day. :)
>
> On Mon, Sep 30, 2024, at 10:09 AM, Štefan Miklošovič wrote:
>
>  "how complex should it be to rip out the chronicle format, insert some
> other well defined and well known, and handle log rolling ourselves".
>
> I think that it will be actually easier to do after CEP-12 is in because
> as I mentioned it does housekeeping of what is there rigth now and
> refactors it a little bit so throwing away the guts of it should be
> isolated only to actual BinLog class and stuff around that which is built
> on top of Chronicle.
>
> "There a reason we can't move forward with CEP-12 w/out addressing the
> chronicle stuff? i.e."
>
> I think there is not any.
>
> "Why would CEP-12 be heavily coupled with chronicle?" - it would not be
> heavier from what is already there for audit of fql. Chronicle here
> basically acts as a sink.
>
> Actually, that patch also makes the implementations of (diagnostic too)
> loggers pluggable  (via coding against an interface and putting that on the
> class path) so people might already write it to whatever they want - even
> to something protobuf-like. If they do not want to use Chronicle as a sink,
> by implementing their own logger, they could just put it wherever they
> want. I think that I forgot to mention this aspect. So the whole solution
> we have is already not hard-wired to Chronicle necessarily. It is just
> something we provide out of the box.
>
> On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie 
> wrote:
>
>
> I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking
> through "how complex should it be to rip out the chronicle format, insert
> some other well defined and well known, and handle log rolling ourselves".
> My preference (which I didn't indicate earlier) would be to have that done
> independently.
>
> There a reason we can't move forward with CEP-12 w/out addressing the
> chronicle stuff? i.e.
>
> I would like to have this resolved because there is CEP-12 I plan to
> deliver and I hit this and I do not want to base that work on something we
> might eventually abandon.
>
> Why would CEP-12 be heavily coupled with chronicle? I would assume you'd
> be able to make light use of the existing logging + log rolling, and then
> someone else could come along and easily rip out chronicle and the rolling
> and add in a different one with minimal code changes?
>
> On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
>
> I don't understand why CEP-12 should be a vehicle for introducing changes
> like that. That is something totally unrelated. I am not going to be the
> one to implement anything beyond CEP-12 and I am not the one who is going
> to replace it either so if we make it a hard requirement for CEP-12 then
> CEP-12 as it is will never be introduced. Just want to be clear about that.
>
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>
> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie 
> wrote:
> >
> > I'd strongly support either rolling the format change into the CEP-12
> proposal or having another CEP for introducing protobuf, spark, etc - some
> kind of more broadly adopted format, and removing chronicle from our stack.
>
> +1, I too would strongly support an open format and removing chronicle.
>
> Kind Regards,
> Brandon
>
>
>
>

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Patrick McFadin]

I'm mentioning it because I was surprised and I feel like I generally have
a finger on the pulse of the project.

I would love to talk about it more and get more community support and
interest.

On Mon, Sep 30, 2024 at 11:01 AM Mick Semb Wever  wrote:

> Agree with Jon, Josh and Patrick here.
>
> This is the type of hidden subproject that will get us into trouble with
> the board/foundation.   I'm sure it's getting enough committer eyeballs,
> and some PMC oversight, but maybe not enough.  Addressing the more material
> points that Jon mentions is the best way to deal with that IMHO.
>
>
>
> On Mon, 30 Sept 2024 at 20:37, Jon Haddad  wrote:
>
>> I think it depends on what lens you're looking at the sidecar through.
>>
>> If you're actively working on it, and pulling it into your own infra,
>> sure.  It's a thing.
>>
>> If you're an outsider?  I have a hard time seeing it.
>>
>> - No documentation as to what it does
>> - No releases
>> - No build instructions
>> - Trying to build using standard gradle commands fails [1]
>> - Included configs don't work out of the box. [2][3]
>> - CEP-1 looks abandonded
>> - The primary reason right now to use it looks to be analytics library,
>> which doesn't work for most teams due to lack of vnode support [4]
>>
>> I think if you were to take a poll of 100 users outside this ML, I'd bet
>> almost every one would agree the sidecar isn't a thing yet, and that's
>> probably more important than if it's actually getting worked on.  I think
>> it has quite a ways to go before it looks to be more than an idea that's
>> incubating.
>>
>> [1] https://issues.apache.org/jira/browse/CASSANDRASC-120
>> [2 https://issues.apache.org/jira/browse/CASSANDRASC-121
>> [3] https://issues.apache.org/jira/browse/CASSANDRASC-122
>> [4] https://issues.apache.org/jira/browse/CASSANDRA-19594
>>
>>
>> On Mon, Sep 30, 2024 at 11:14 AM Josh McKenzie 
>> wrote:
>>
>>> The CEP for the sidecar has stalled. The sidecar itself is very much
>>> alive and a thing.
>>>
>>> CEP != artifact.
>>>
>>> We should definitely clean that up though.
>>>
>>> On Mon, Sep 30, 2024, at 10:59 AM, Dinesh Joshi wrote:
>>>
>>> Patrick, could you please elaborate? The Sidecar has been a thing for a
>>> while now.
>>>
>>> On Mon, Sep 30, 2024 at 7:51 AM Patrick McFadin 
>>> wrote:
>>>
>>> I made the mistake of asking two things in one email.
>>>
>>> First thing I asked. Sidecar? Stalled CEP so why is this being talked
>>> about like this is a thing?
>>>
>>> On Mon, Sep 30, 2024 at 7:21 AM Benedict  wrote:
>>>
>>>
>>> Sorry Bernardo, you may have misunderstood me. I don’t have any
>>> concerns, I was suggesting a possible future scenario where CDC for Kafka
>>> via sidecar is changed to use a hypothetical future topic subscription
>>> service provided by C*. It was meant to show that this CEP may be easily
>>> decoupled from any future evolution in this area.
>>>
>>>
>>> On 30 Sep 2024, at 14:58, Bernardo Botella 
>>> wrote:
>>>
>>> Thanks everyone for the comments.
>>>
>>>
>>> Patrick:
>>> The proposal includes a “best effort” approach for deduplication (some
>>> details can be found on the Digest class comments on the PR here
>>> https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
>>>  ).
>>> That alone won’t eliminate all the duplicates, but as Josh points out, it
>>> moves the line to something way easier to handle for consumers, and
>>> definitely on the direction we should aim towards. Accord is definitely
>>> something this contribution will benefit from, that will move that line
>>> even further.
>>>
>>> Benedict:
>>> If I understand it correctly, your concern is that Kafka is somewhat the
>>> hardcoded option for a CDC stream being published? The proposal introduces
>>> a concept of data sources and sinks (
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks)
>>> being kafka the first implemented data sink. That means that the actual
>>> Kafka output should (will) be something pluggable.
>>>
>>>
>>>
>>> On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:
>>>
>>> I don't see much on how this would be handled other than "left to the
>>> end user to figure out."
>>>
>>> My immediate thought when I read that was "Yes. But it's moving where we
>>> draw the line of 'left to the end user to figure out' *much further* than
>>> it was before".
>>>
>>> This should only be necessary in edge cases w/extended severe degraded
>>> availability where you can't hit QUORUM w/this design. So we go from
>>> "De-dupe literally everything o ye' user" to "de-dupe a small fraction of a
>>> % of the time when things really go off the rails".
>>>
>>> It still leaves the burden of processing potential duplicates
>>> downstream, so some *complexity* burden on the users remains if they
>>> have no tolerance for processing duplicate messages, however the underlying
>>> mach

[RELEASE] Apache Cassandra 5.0.1 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 5.0.1.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 5.0 series. As always, please
pay attention to the release notes[2] and Let us know[3] if you were to
encounter any problem.

==
[WARNING]
This release introduces safeguards and observability into possible data
loss scenarios when nodes have a  divergent view of the cluster. This
happens around edge-cases on unsafe bootstrapping, decommissions, or when a
node has a corrupted topology. Two configuration options have been added:
`log_out_of_token_range_requests` and
 `reject_out_of_token_range_requests`, both enabled by default. The former
will make nodes log requests they receive that don't belong in their
current or pending token ranges. The latter will reject those requests,
which prevents any eventual data loss that can occur but may also incur
small windows of degraded availability during range movements. See
CASSANDRA-13704 for further details.
==

Enjoy!

[1]: CHANGES.txt
https://github.com/apache/cassandra/blob/cassandra-5.0.1/CHANGES.txt
[2]: NEWS.txt
https://github.com/apache/cassandra/blob/cassandra-5.0.1/NEWS.txt
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Josh McKenzie]

LGTM

On Mon, Sep 30, 2024, at 2:17 PM, Štefan Miklošovič wrote:
> Okay, that works, so summary:
> 
> 1) CEP-12 will be delivered WITHOUT interacting with Chronicle Queues, 
> something else will be used (most probably just a simple logger logging to a 
> text file, that is just enough for something like diagnostic logs which are 
> logged very sparsely)
> 
> 2) We will NOT use Chronicle in the future for any new implementations / 
> solutions from now on.
> 
> 3) We WILL be replacing loggers based on Chronicle Queues.
> 
> 4) A new implementation replacing Chronicle Queues SHOULD take into account a 
> heterogeneous environment when it comes to consumers of the events logged, 
> being language-agnostic or supporting an open format which enables that.
> 
> On Mon, Sep 30, 2024 at 5:07 PM Josh McKenzie  wrote:
>> __
>> I say we do CEP-12 w/out using chronicle and have a follow up JIRA to 
>> replace chronicle w/something else.
>> 
>> Seems like there's a reasonable consensus around at least that subset of 
>> things given the discussion here. As to what form that something else takes, 
>> well, that's a topic for another day. :)
>> 
>> On Mon, Sep 30, 2024, at 10:09 AM, Štefan Miklošovič wrote:
>>>  "how complex should it be to rip out the chronicle format, insert some 
>>> other well defined and well known, and handle log rolling ourselves".
>>>  
>>> I think that it will be actually easier to do after CEP-12 is in because as 
>>> I mentioned it does housekeeping of what is there rigth now and refactors 
>>> it a little bit so throwing away the guts of it should be isolated only to 
>>> actual BinLog class and stuff around that which is built on top of 
>>> Chronicle.
>>> 
>>> "There a reason we can't move forward with CEP-12 w/out addressing the 
>>> chronicle stuff? i.e."
>>> 
>>> I think there is not any.
>>> 
>>> "Why would CEP-12 be heavily coupled with chronicle?" - it would not be 
>>> heavier from what is already there for audit of fql. Chronicle here 
>>> basically acts as a sink.
>>> 
>>> Actually, that patch also makes the implementations of (diagnostic too) 
>>> loggers pluggable  (via coding against an interface and putting that on the 
>>> class path) so people might already write it to whatever they want - even 
>>> to something protobuf-like. If they do not want to use Chronicle as a sink, 
>>> by implementing their own logger, they could just put it wherever they 
>>> want. I think that I forgot to mention this aspect. So the whole solution 
>>> we have is already not hard-wired to Chronicle necessarily. It is just 
>>> something we provide out of the box.
>>> 
>>> On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie  wrote:
 __
 I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking 
 through "how complex should it be to rip out the chronicle format, insert 
 some other well defined and well known, and handle log rolling ourselves". 
 My preference (which I didn't indicate earlier) would be to have that done 
 independently.
 
 There a reason we can't move forward with CEP-12 w/out addressing the 
 chronicle stuff? i.e.
> I would like to have this resolved because there is CEP-12 I plan to 
> deliver and I hit this and I do not want to base that work on something 
> we might eventually abandon.
 Why would CEP-12 be heavily coupled with chronicle? I would assume you'd 
 be able to make light use of the existing logging + log rolling, and then 
 someone else could come along and easily rip out chronicle and the rolling 
 and add in a different one with minimal code changes?
 
 On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
> I don't understand why CEP-12 should be a vehicle for introducing changes 
> like that. That is something totally unrelated. I am not going to be the 
> one to implement anything beyond CEP-12 and I am not the one who is going 
> to replace it either so if we make it a hard requirement for CEP-12 then 
> CEP-12 as it is will never be introduced. Just want to be clear about 
> that.
> 
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  
>> wrote:
>> >
>> > I'd strongly support either rolling the format change into the CEP-12 
>> > proposal or having another CEP for introducing protobuf, spark, etc - 
>> > some kind of more broadly adopted format, and removing chronicle from 
>> > our stack.
>> 
>> +1, I too would strongly support an open format and removing chronicle.
>> 
>> Kind Regards,
>> Brandon
 
>> 

Re: [DISCUSS] CEP-44: Kafka integration for Cassandra CDC using Sidecar

[Mick Semb Wever]

Agree with Jon, Josh and Patrick here.

This is the type of hidden subproject that will get us into trouble with
the board/foundation.   I'm sure it's getting enough committer eyeballs,
and some PMC oversight, but maybe not enough.  Addressing the more material
points that Jon mentions is the best way to deal with that IMHO.



On Mon, 30 Sept 2024 at 20:37, Jon Haddad  wrote:

> I think it depends on what lens you're looking at the sidecar through.
>
> If you're actively working on it, and pulling it into your own infra,
> sure.  It's a thing.
>
> If you're an outsider?  I have a hard time seeing it.
>
> - No documentation as to what it does
> - No releases
> - No build instructions
> - Trying to build using standard gradle commands fails [1]
> - Included configs don't work out of the box. [2][3]
> - CEP-1 looks abandonded
> - The primary reason right now to use it looks to be analytics library,
> which doesn't work for most teams due to lack of vnode support [4]
>
> I think if you were to take a poll of 100 users outside this ML, I'd bet
> almost every one would agree the sidecar isn't a thing yet, and that's
> probably more important than if it's actually getting worked on.  I think
> it has quite a ways to go before it looks to be more than an idea that's
> incubating.
>
> [1] https://issues.apache.org/jira/browse/CASSANDRASC-120
> [2 https://issues.apache.org/jira/browse/CASSANDRASC-121
> [3] https://issues.apache.org/jira/browse/CASSANDRASC-122
> [4] https://issues.apache.org/jira/browse/CASSANDRA-19594
>
>
> On Mon, Sep 30, 2024 at 11:14 AM Josh McKenzie 
> wrote:
>
>> The CEP for the sidecar has stalled. The sidecar itself is very much
>> alive and a thing.
>>
>> CEP != artifact.
>>
>> We should definitely clean that up though.
>>
>> On Mon, Sep 30, 2024, at 10:59 AM, Dinesh Joshi wrote:
>>
>> Patrick, could you please elaborate? The Sidecar has been a thing for a
>> while now.
>>
>> On Mon, Sep 30, 2024 at 7:51 AM Patrick McFadin 
>> wrote:
>>
>> I made the mistake of asking two things in one email.
>>
>> First thing I asked. Sidecar? Stalled CEP so why is this being talked
>> about like this is a thing?
>>
>> On Mon, Sep 30, 2024 at 7:21 AM Benedict  wrote:
>>
>>
>> Sorry Bernardo, you may have misunderstood me. I don’t have any concerns,
>> I was suggesting a possible future scenario where CDC for Kafka via sidecar
>> is changed to use a hypothetical future topic subscription service provided
>> by C*. It was meant to show that this CEP may be easily decoupled from any
>> future evolution in this area.
>>
>>
>> On 30 Sep 2024, at 14:58, Bernardo Botella 
>> wrote:
>>
>> Thanks everyone for the comments.
>>
>>
>> Patrick:
>> The proposal includes a “best effort” approach for deduplication (some
>> details can be found on the Digest class comments on the PR here
>> https://github.com/apache/cassandra-analytics/pull/87/files#diff-3a09caecc1da13419d92cde56a7cfc7d253faac08182e6c2768b3d32c015de82R185-R193
>>  ).
>> That alone won’t eliminate all the duplicates, but as Josh points out, it
>> moves the line to something way easier to handle for consumers, and
>> definitely on the direction we should aim towards. Accord is definitely
>> something this contribution will benefit from, that will move that line
>> even further.
>>
>> Benedict:
>> If I understand it correctly, your concern is that Kafka is somewhat the
>> hardcoded option for a CDC stream being published? The proposal introduces
>> a concept of data sources and sinks (
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488575#CEP44:KafkaintegrationforCassandraCDCusingSidecar-SourcesandSinks)
>> being kafka the first implemented data sink. That means that the actual
>> Kafka output should (will) be something pluggable.
>>
>>
>>
>> On Sep 30, 2024, at 5:43 AM, Josh McKenzie  wrote:
>>
>> I don't see much on how this would be handled other than "left to the end
>> user to figure out."
>>
>> My immediate thought when I read that was "Yes. But it's moving where we
>> draw the line of 'left to the end user to figure out' *much further* than
>> it was before".
>>
>> This should only be necessary in edge cases w/extended severe degraded
>> availability where you can't hit QUORUM w/this design. So we go from
>> "De-dupe literally everything o ye' user" to "de-dupe a small fraction of a
>> % of the time when things really go off the rails".
>>
>> It still leaves the burden of processing potential duplicates downstream,
>> so some *complexity* burden on the users remains if they have no
>> tolerance for processing duplicate messages, however the underlying machine
>> resource utilization (from "dedupe everything" to "dedupe a small % of
>> things") is pretty massively shifted by this design change. That, and using
>> the hash of the mutation the way the extended design does is something a
>> downstream consumer could also do on their side to ensure anything that
>> came in past the drop-off window wasn't already seen. So not *too* p

Re: CEP-15: Accord status

[Patrick McFadin]

I hear LLMs are good for this. Just something I saw on YouTube :D

On Mon, Sep 30, 2024 at 5:40 AM Štefan Miklošovič 
wrote:

> I think there was some discussion about putting that all to the website
> which never materialized.
>
> On Mon, Sep 30, 2024 at 2:37 PM Josh McKenzie 
> wrote:
>
>> Today I learned… I had no clue we had markdown files in src/java…
>>
>> Discoverability issues in our codebase?
>>
>> Well I never.
>>
>> ;)
>>
>> On Sat, Sep 28, 2024, at 10:39 PM, David Capwell wrote:
>>
>> Today I learned… I had no clue we had markdown files in src/java…
>>
>> $ find src/ -name '*.md'
>> src//java/org/apache/cassandra/io/sstable/SSTable_API.md
>> src//java/org/apache/cassandra/io/sstable/format/bti/BtiFormat.md
>> src//java/org/apache/cassandra/utils/bytecomparable/ByteComparable.md
>> src//java/org/apache/cassandra/tcm/TCM_implementation.md
>> src//java/org/apache/cassandra/tcm/TransactionalClusterMetadata.md
>> src//java/org/apache/cassandra/db/memtable/Memtable_API.md
>> src//java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.md
>> src//java/org/apache/cassandra/db/tries/InMemoryTrie.md
>> src//java/org/apache/cassandra/db/tries/Trie.md
>> src//java/org/apache/cassandra/index/sai/README.md
>> src//java/org/apache/cassandra/service/paxos/Paxos.md
>>
>>
>>
>> We don’t have one at the moment but it would be good to get that in.  At
>> a high level there are a few key classes
>>
>> 1) org.apache.cassandra.cql3.statements.TransactionStatement - this class
>> handles BEGIN TRANSACTION in CQL
>> 2) org.apache.cassandra.service.consensus.TransactionalMode - this is a
>> table property and dictates what is allowed for the table.  If off accord
>> transactions are not allowed, if “full” normal read/write get migrated to
>> Accord (and you can still use BEGIN TRANSACTION)
>> 3) org.apache.cassandra.service.accord.AccordService - the global static
>> instance that lets Cassandra call Accord stuff
>>
>> On Sep 27, 2024, at 7:20 AM, Paulo Motta  wrote:
>>
>> Thanks all for the work on this epic!
>>
>> Is there an implementation summary guide similar to guide_8099.md [1]
>> that can help reviewers not involved with the effort navigate through the
>> code ? It would be great to have it if this is not already available or
>> being planned. There's a similar one though much smaller in scope for
>> memtable API on [2].
>>
>> [1] -
>> https://github.com/apache/cassandra/blob/cassandra-3.0.0-rc2/guide_8099.md
>>
>> [2] -
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/memtable/Memtable_API.md
>>
>> On Fri, Sep 27, 2024 at 8:09 AM Benedict Elliott Smith <
>> bened...@apache.org> wrote:
>>
>> If you exclude test changes, there’s < 50k added and ~2k removed. This
>> works out to ~7% of the scale of 8099 for lines modified, if this is the
>> benchmark for disruption.
>>
>> Altogether, this is a very small patch from the perspective of the
>> existing codebase. Probably doesn’t even come close to the top 10.
>>
>> Conversely, for new standalone features, this is likely the most complex
>> thing we have ever merged to the project. But, it is off by default, and
>> the risk to deployments therefore is very minimal.
>>
>> Regarding how parties can engage, I think if we’re honest history shows
>> that engagement will be minimal. There have after all been several touch
>> points, and none have materialised into really significant engagement. This
>> is just the reality of everyone having their own pressures - at the end of
>> the day, changes happen and the community adapts. But, we are here to
>> answer any questions - as we have been throughout the development of the
>> work in the open.
>>
>>
>>
>> On 20 Sep 2024, at 22:08, Josh McKenzie  wrote:
>>
>> This presents an opportune moment for those interested to review the code.
>> ...
>> +88,341 −7,341
>> 1003 Files changed
>>
>>
>> O.o
>> This is... *very large*. If we use CASSANDRA-8099 as our "banana for
>> scale":
>>
>> 645 files changed, 49381 insertions(+), 42227 deletions(-)
>>
>>
>> To be clear - I don't think we collectively should be worried about
>> disruption from this patch since:
>>
>>1. Each commit (or the vast majority?) has already been reviewed by
>>>= 1 other committer
>>2. 7.3k deletions is a lot less than 42k
>>3. We now have fuzzing, property based testing, and the simulator
>>4. Most of this code is additive
>>
>> How would you recommend interested parties engage with reviewing this
>> behemoth? Or perhaps subsections of it or key areas to familiarize
>> themselves with the structure?
>>
>> On Fri, Sep 20, 2024, at 12:17 PM, David Capwell wrote:
>>
>> Recently, we rebased against the trunk branch, ensuring that the accord
>> branch is now in sync with the latest trunk version. This presents an
>> opportune moment for those interested to review the code.
>>
>> We have a pending pull request 
>> (*https://github.com/apache/cassandra/pull/3552
>>