Memory allocation in native transport: Netty allocator vs networking_cache
Hi all, I am analyzing how memory allocation is organized in Cassandra (4.x-5.x) and have a question about memory allocators usage in Apache Cassandra code. Some time ago Cassandra-specific networking_cache has been introduced ( https://issues.apache.org/jira/browse/CASSANDRA-15066 + https://issues.apache.org/jira/browse/CASSANDRA-15229) and configured as an allocator for Netty channels in internode messaging as well as in native transport: org.apache.cassandra.net.LocalBufferPoolAllocator org.apache.cassandra.transport.PipelineConfigurator#configureModernPipeline org.apache.cassandra.net.InboundConnectionInitiator.Initializer#initChannel org.apache.cassandra.net.OutboundConnectionInitiator#createBootstrap At the same time there are few places in native transport code where the original Netty PooledByteBufAllocator (org.apache.cassandra.transport.CBUtil#allocator instance) is still in use, like: org.apache.cassandra.transport.Message#encode org.apache.cassandra.transport.Compressor.LZ4Compressor org.apache.cassandra.transport.Compressor.SnappyCompressor Can somebody help to understand: is the usage of different allocators in native transport logic intentional or the original idea was to migrate fully to the Cassandra networking_cache as a Netty allocator? Thank you, Dmitry -- Dmitry Konstantinov
[CASSANDRA-19651][Review] idealCLWriteLatency metric reports the worst response time instead of the time when ideal CL is satisfied
Hello All, I recently submitted a patch what addresses a bug in idealCLWriteLatency metric calculation: https://issues.apache.org/jira/browse/CASSANDRA-19651 Could someone help with reviewing it? Thank you for your time! -- Dmitry
purpose of queryStates map in ServerConnection
Hello, Could you please help to understand the purpose of a particular part of code in Cassandra codebase: org.apache.cassandra.transport.ServerConnection#getQueryState https://github.com/apache/cassandra/blob/cassandra-3.11.6/src/java/org/apache/cassandra/transport/ServerConnection.java#L46 I noticed it originally in a profiler output: [image: image.png] For each CQL connection we have here cached QueryState instances per each streamId value. This map is only dropped/garbage collected when ServerCollection is released. Can we just create a new QueryState for each request? Regards, Dmitry -- Dmitry Konstantinov
[Committer/reviewer needed][CASSJAVA-4] UDT codec decode is too restrictive in decoding of unknown fields
Hello All, Some time ago I have submitted a patch what makes UDT codec tolerant to extra unknown fields at the tail of UDT to support live schema upgrades: https://issues.apache.org/jira/browse/CASSJAVA-4 Could someone help with reviewing it? Thank you for your time! -- Dmitry Konstantinov
Re: [DISCUSS] Replacement of SSTable's partition cardinality implementation from stream-lib to Apache Datasketches
Hi Brain, I wanted it to be created under https://cwiki.apache.org/confluence/display/CASSANDRA/Discussion but it looks like I do not have grants to add a page there and Confluence automatically selected this space to store the page. I do not have permission to move it too :-( Can I get grants to create pages under https://cwiki.apache.org/confluence/display/CASSANDRA/ ? Thank you, Dmitry On Fri, 3 Jan 2025 at 14:12, Brian Proffitt wrote: > Dmitry: > > You are using a section of the Confluence wiki that is dedicated to > Community Over Code, the Apache Conference. Please move that page to a more > appropriate part of the Apache wiki as soon as you can. > > Thanks! > BKP > > On 2025/01/03 13:55:49 Dmitry Konstantinov wrote: > > I have summarized information from this mail thread to > > > https://cwiki.apache.org/confluence/display/COC/SSTable%27s+partition+cardinality+implementation > > Probably later it can be transformed to a CEP.. > > Regarding experience of DataSketches library's authors and publications > > here there is a good summary in Background section: > > > https://cwiki.apache.org/confluence/display/INCUBATOR/DataSketchesProposal > > . It looks good.. > > > > On Fri, 3 Jan 2025 at 13:06, Štefan Miklošovič > > wrote: > > > > > Right ... that sounds reasonable. Let's "sleep on it" for a while. It > is > > > not something which is urgent to deal with right now but I find myself > > > quite often to identify the functionality where we go to the disk more > > > often than necessary and this was next on the list to take a look at > > > reading CASSANDRA-13338. So I took a look ... and here we are. > > > > > > If you guys go to bump SSTable version in 5.1 / 6.0, this change might > be > > > just shipped with that too. > > > > > > On Fri, Jan 3, 2025 at 1:47 PM Benedict wrote: > > > > > >> I’ve had a quick skim of the data sketches library, and it does seem > to > > >> have made some more efficient decisions in its design than > clearspring, > > >> appears to maybe support off-heap representations, and has reasonably > good > > >> documentation about the theoretical properties of the sketches. The > chair > > >> of the project is a published author on the topic, and the library has > > >> newer algorithms for cardinality estimation than HLL. > > >> > > >> So, honestly, it might not be a bad idea to (carefully) consider a > > >> migration, even if the current library isn’t broken for our needs. > > >> > > >> It would not be high up my priority list for the project, but I would > > >> support it if it scratches someone’s itch. > > >> > > >> On 3 Jan 2025, at 12:16, Štefan Miklošovič > > >> wrote: > > >> > > >> > > >> Okay ... first problems. > > >> > > >> These 2000 bytes I have mentioned in my response to Chris were indeed > > >> correct, but that was with Datasketches and the main parameter for > Hall > > >> Sketch (DEFAULT_LG_K) was 12. When I changed that to 13 to match what > we > > >> currently have in Cassandra with Clearspring, that doubled the size to > > >> ~4000 bytes. > > >> > > >> When we do not use Datasketches, what Clearspring generates is about > > >> ~5000 bytes for the array itself but that array is wrapped into an > > >> ICardinality object of Clearspring and we need that object in order to > > >> merge another ICardinality into that. So, we would need to cache this > > >> ICardinality object instead of just an array itself. If we don't cache > > >> whole ICardinality, we would then need to do basically what > > >> CompactionMetadata.CompactionMetadataSerializer.deserialize is doing > which > > >> would allocate a lot / often (ICardinality cardinality = > > >> HyperLogLogPlus.Builder.build(that_cached_array)). > > >> > > >> To avoid the allocations every time we compute, we would just cache > that > > >> whole ICardinality of Clearspring, but that whole object measures like > > >> 11/12 KB. So even 10k tables would occupy like 100MB. 50k tables > 500MB. > > >> That is becoming quite a problem. > > >> > > >> On the other hand, HllSketch of Datasketches, array included, adds > > >> minimal overhead. Like an array has 5000 bytes and the whole object > like > > >> 5500. You got the idea ... > > >> > &
Re: [DISCUSS] Replacement of SSTable's partition cardinality implementation from stream-lib to Apache Datasketches
Thank you, Brandon! I have moved the page to https://cwiki.apache.org/confluence/display/CASSANDRA/SSTable%27s+partition+cardinality+implementation On Fri, 3 Jan 2025 at 14:45, Brandon Williams wrote: > I've granted access to the account "Dmitry Konstantinov (netudima)" > > Kind Regards, > Brandon > > On Fri, Jan 3, 2025 at 8:18 AM Dmitry Konstantinov > wrote: > > > > Hi Brain, I wanted it to be created under > https://cwiki.apache.org/confluence/display/CASSANDRA/Discussion but it > looks like I do not have grants to add a page there and Confluence > automatically selected this space to store the page. > > I do not have permission to move it too :-( > > Can I get grants to create pages under > https://cwiki.apache.org/confluence/display/CASSANDRA/ ? > > > > Thank you, > > Dmitry > > > > On Fri, 3 Jan 2025 at 14:12, Brian Proffitt wrote: > >> > >> Dmitry: > >> > >> You are using a section of the Confluence wiki that is dedicated to > Community Over Code, the Apache Conference. Please move that page to a more > appropriate part of the Apache wiki as soon as you can. > >> > >> Thanks! > >> BKP > >> > >> On 2025/01/03 13:55:49 Dmitry Konstantinov wrote: > >> > I have summarized information from this mail thread to > >> > > https://cwiki.apache.org/confluence/display/COC/SSTable%27s+partition+cardinality+implementation > >> > Probably later it can be transformed to a CEP.. > >> > Regarding experience of DataSketches library's authors and > publications > >> > here there is a good summary in Background section: > >> > > https://cwiki.apache.org/confluence/display/INCUBATOR/DataSketchesProposal > >> > . It looks good.. > >> > > >> > On Fri, 3 Jan 2025 at 13:06, Štefan Miklošovič < > smikloso...@apache.org> > >> > wrote: > >> > > >> > > Right ... that sounds reasonable. Let's "sleep on it" for a while. > It is > >> > > not something which is urgent to deal with right now but I find > myself > >> > > quite often to identify the functionality where we go to the disk > more > >> > > often than necessary and this was next on the list to take a look at > >> > > reading CASSANDRA-13338. So I took a look ... and here we are. > >> > > > >> > > If you guys go to bump SSTable version in 5.1 / 6.0, this change > might be > >> > > just shipped with that too. > >> > > > >> > > On Fri, Jan 3, 2025 at 1:47 PM Benedict > wrote: > >> > > > >> > >> I’ve had a quick skim of the data sketches library, and it does > seem to > >> > >> have made some more efficient decisions in its design than > clearspring, > >> > >> appears to maybe support off-heap representations, and has > reasonably good > >> > >> documentation about the theoretical properties of the sketches. > The chair > >> > >> of the project is a published author on the topic, and the library > has > >> > >> newer algorithms for cardinality estimation than HLL. > >> > >> > >> > >> So, honestly, it might not be a bad idea to (carefully) consider a > >> > >> migration, even if the current library isn’t broken for our needs. > >> > >> > >> > >> It would not be high up my priority list for the project, but I > would > >> > >> support it if it scratches someone’s itch. > >> > >> > >> > >> On 3 Jan 2025, at 12:16, Štefan Miklošovič > > >> > >> wrote: > >> > >> > >> > >> > >> > >> Okay ... first problems. > >> > >> > >> > >> These 2000 bytes I have mentioned in my response to Chris were > indeed > >> > >> correct, but that was with Datasketches and the main parameter for > Hall > >> > >> Sketch (DEFAULT_LG_K) was 12. When I changed that to 13 to match > what we > >> > >> currently have in Cassandra with Clearspring, that doubled the > size to > >> > >> ~4000 bytes. > >> > >> > >> > >> When we do not use Datasketches, what Clearspring generates is > about > >> > >> ~5000 bytes for the array itself but that array is wrapped into an > >> > >> ICardinality object of Clearspring and we need that object in > order t
Re: [DISCUSS] Replacement of SSTable's partition cardinality implementation from stream-lib to Apache Datasketches
CS_Hllpp.html >>>> >>>> On Thu, Jan 2, 2025 at 5:03 PM Benedict wrote: >>>> >>>>> I am strongly opposed to updating libraries simply for the sake of it. >>>>> Something like HLL does not need much ongoing maintenance if it works. >>>>> We’re simply asking for extra work and bugs by switching, and some risk >>>>> without understanding the quality control for the new library project’s >>>>> releases. >>>>> >>>>> That said, I was not very impressed with the clear spring library when >>>>> I looked at it, so I would be open to a stronger argument about data >>>>> sketches being superior otherwise in a way that matters to us. >>>>> >>>>> If we are to replace the library, we should at the very least do >>>>> proper due diligence by reviewing the new library’s implementation(s) >>>>> ourselves. We cannot simply assume the new library behaves well for our >>>>> use >>>>> cases, or is well maintained. >>>>> >>>>> We should also not use the fallback intersection method, as this would >>>>> represent a regression to compaction on upgrade. We should really convert >>>>> from one HLL to another. >>>>> >>>>> The proposal to reduce allocations appears to be orthogonal to this >>>>> library, so let’s separate out that discussion? If there’s evidence this >>>>> library alone improves the memory profile let’s discuss that. >>>>> >>>>> >>>>> On 2 Jan 2025, at 15:26, Chris Lohfink wrote: >>>>> >>>>> >>>>> I think switching to datasketches is a good idea first off simply >>>>> because of the lack of maintenance and improvements from clearspring. I am >>>>> however, am not sold that it will actually improve anything significantly. >>>>> Caches might help on small cases, but those small cases probably are not >>>>> actually impacted. In the large cases the caches cost more in complexity, >>>>> memory, and ultimately wont matter when theres 50k sstables and the cache >>>>> holds 1k so everythings hitting disk anyway. >>>>> >>>>> The 5% is missing some relevant information like what the allocation >>>>> rate was, how many tables there are etc. On an idle system thats >>>>> meaningless, if there were 5gb/s allocations of reads/writes happening at >>>>> the time thats huge. >>>>> >>>>> On Thu, Jan 2, 2025 at 8:42 AM Štefan Miklošovič < >>>>> smikloso...@apache.org> wrote: >>>>> >>>>>> Interesting, thanks for this. Well ... 5% here, 5% there ... it >>>>>> compounds. I think it is worth trying to do something with this. Would be >>>>>> great if you were part of this effort! >>>>>> >>>>>> On Thu, Jan 2, 2025 at 3:38 PM Dmitry Konstantinov < >>>>>> netud...@gmail.com> wrote: >>>>>> >>>>>>> I have seen this place in async profiler memory allocation profile >>>>>>> on one of production environments some time ago, it was visible but not >>>>>>> dramatic, about 5% of allocations: >>>>>>> >>>>>>> >>>>>>> The amount of overhead also depends on a metric collection frequency >>>>>>> (in my case it was once per 60 seconds) >>>>>>> >>>>>>> Regards, >>>>>>> Dmitry >>>>>>> >>>>>>> On Thu, 2 Jan 2025 at 14:21, Štefan Miklošovič < >>>>>>> smikloso...@apache.org> wrote: >>>>>>> >>>>>>>> Indeed, I plan to measure it and compare, maybe some bench test >>>>>>>> would be cool to add .. >>>>>>>> >>>>>>>> I strongly suspect that the primary reason for the slowness (if it >>>>>>>> is verified to be true) is us going to the disk every time and reading >>>>>>>> stats for every SSTable all over again. >>>>>>>> >>>>>>>> While datasketches say that it is way faster to update (1), we are >>>>>>>> living in a realm of nanoseconds here and I don't think that itself >>>>>>>> would >>>>>>>>
[Reviewer required] CASSANDRA-20132 - Add metric and tracing event for scanned purgeable tombstones
Hi, Can somebody help with reviewing of https://issues.apache.org/jira/browse/CASSANDRA-20132. When tombstones are expired they become almost invisible from a monitoring point view: you do not see them in metrics and tracing except a latency impact. I have observed such cases in production when compaction did not remove expired tombstones fast enough and in some cases we got a lot of them for a partition, with a bad latency as a result. It was not that easy to find that you have them (they are visible now only in file system calls or via sstabledump analysis). The improvement adds visibility here. Regards, Dmitry
Re: [VOTE] CEP-45: Mutation Tracking
+1 (nb) On Tue, 4 Feb 2025 at 22:00, Abe Ratnofsky wrote: > +1 (nb) > -- Dmitry Konstantinov
Re: [DISCUSS] CEP-45: Mutation Tracking
Hello Blake, thank you a lot for sharing the CEP, it looks really promising and should address many of the current pain points! I have a few questions to clarify: 1) Is a single separate commit log expected to be created for all tables with the new replication type? 2) What is a granularity of storing mutation ids in memtable, is it per cell? 3) If we update the same row multiple times while it is in a memtable - are all mutation ids appended to a kind of collection? 4) What is the expected size of a single id? 5) Do we plan to support multi-table batches (single or multi-partition) for this replication type? Thank you, Dmitry On Wed, 8 Jan 2025 at 19:22, Blake Eggleston wrote: > Hello dev@, > > We'd like to propose CEP-45: Mutation Tracking for adoption by the > community. CEP-45 proposes adding a replication mechanism to track and > reconcile individual mutations, as well as processes to actively reconcile > missing mutations. > > For keyspaces with mutation tracking enabled, the immediate benefits of > this CEP are: > * reduced replication lag with a continuous background reconciliation > process > * eliminate the disk load caused by repair merkle tree calculation > * eliminate repair overstreaming > * reduce disk load of reads on cluster to close to 1/CL > * fix longstanding mutation atomicity issues caused by read repair and > short read protection > > Additionally, although it's outside the scope of this CEP, mutation > tracking would enable: > * completion of witness replicas / transient replication, making the > feature usable for all workloads > * lightweight witness only datacenters > > The CEP is linked here: > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-45%3A+Mutation+Tracking, > but please keep the discussion on the dev list. > > Thanks! > > Blake Eggleston > -- Dmitry Konstantinov
Re: What branches should perf fixes be targeting
releases are for bug fixes not perf improvements. >> >> >> On Jan 21, 2025, at 9:10 PM, Jordan West wrote: >> >> >> Hi folks, >> >> A topic that’s come up recently is what branches are valid targets for >> performance improvements. Should they only go into trunk? This has come up >> in the context of BTI improvements, Dmitry’s work on reducing object >> overhead, and my work on CASSANDRA-15452. >> >> We currently have guidelines published: >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199530302#Patching,versioning,andLTSreleases-Wheretoapplypatches. >> But there’s no explicit discussion of how to handle performance >> improvements. We tend to discuss whether they’re “bugfixes”. >> >> I’d like to discuss whether performance improvements should target more >> than just trunk. I believe they should target every active branch because >> performance is a major selling point of Cassandra. It’s not practical to >> ask users to upgrade major versions for simple performance wins. A major >> version can be deployed for years, especially when the next one has major >> changes. But we shouldn’t target non-supported major versions, either. >> Also, there will be exceptions: patches that are too large, invasive, >> risky, or complicated to backport. For these, we rely on the contributor >> and reviewer’s judgment and the mailing list. So, I’m proposing an >> allowance to backport to active branches, not a requirement to merge them. >> >> I’m curious to hear your thoughts. >> Jordan >> >> -- Dmitry Konstantinov
Re: Patrick McFadin joins the PMC
Congratulations! On Wed, 22 Jan 2025 at 16:36, Russell Spitzer wrote: > Congratulations! > > On Wed, Jan 22, 2025 at 10:30 AM Ekaterina Dimitrova < > e.dimitr...@gmail.com> wrote: > >> Congratulations, Patrick 🎉 >> >> On Wed, 22 Jan 2025 at 11:05, Jordan West wrote: >> >>> The PMC's members are pleased to announce that Patrick McFadin has accepted >>> an invitation to become a PMC member. >>> >>> Thanks a lot, Patrick, for everything you have done for the project all >>> these years. >>> >>> Congratulations and welcome!! >>> >>> The Apache Cassandra PMC >>> >> -- Dmitry Konstantinov
Re: What branches should perf fixes be targeting
Hi Stefan, Thank you a lot for the detailed feedback! Few comments: >> I think this is already the case, more or less. We are not doing perf changes in older branches. Yes, I understand the idea about stability of older branches, the primary issue for me is that if I contribute even a small improvement to trunk - I cannot really use it for a long time (except having it in my own fork), because there is no release to get it back for me or anybody else.. >> Maybe it would be better to make the upgrading process as smooth as possible so respective businesses are open to upgrade their clusters in a more frequent manner. About the upgrade process: my personal experience (3.0.x -> 3.11.x -> 4.0.x -> 4.1.x), the upgrade in Cassandra is positive (I suppose the autotests which test it are really helpful), I have not experienced any serious issues with it. I suppose the majority of time when people have an issue with upgrades is due to delaying them for too long and staying on very old unsupported versions till the last moment. >> Cassandra is not JDK. We need to fix bugs in older branches we said we support Regarding the necessity to support the older branches it is the same story for JDK: they now support and fix bugs in JDK8, JDK11, JDK17 and JDK 21 as LTS versions and JDK23 as the latest release while developing and releasing JDK24 now. Another example, Postgres does a major release every year: https://www.postgresql.org/support/versioning/ and supports the last 5 major versions. >> please keep in mind that there are people behind the releases who are spending time on that. Yes, as I already mentioned, I really thank you to Brandon and Mick for doing it! It is hard, exhausting and not the most exciting work to do. Please contact me if I can help somehow with it, like checking and fixing CI test failures(I've already done it for a while) / doing some scripting/ etc. I have a hypothesis (maybe I am completely wrong here) that actually the low interest in the releasing process is somehow related to having a Cassandra fork by many contributors, so there is no big demand for regular mainline releases if you have them in a fork.. Regards, Dmitry On Thu, 23 Jan 2025 at 12:30, Štefan Miklošovič wrote: > I think the current guidelines are sensible. > > Going through your suggestions: > > 1) I think this is already the case, more or less. We are not doing perf > changes in older branches. This is what we see in CASSANDRA-19429, a user > reported that it is a performance improvement, and most probably he is > right, but I am hesitant to refactor / introduce changes into older > branches. > > Cassandra has a lot of inertia, we can not mess with what works even > performance improvements are appealing. Maybe it would be better to make > the upgrading process as smooth as possible so respective businesses are > open to upgrade their clusters in a more frequent manner. > > 2) Well, but Cassandra is not JDK. We need to fix bugs in older branches > we said we support. This is again related to inertia Cassandra has as a > database. Bug fixes are always welcome, especially if there is 0 risk > deploying it. > > What particularly resonates with me is your wording "more frequent and > predictable". Well ... I understand it would be the most ideal outcome, but > please keep in mind that there are people behind the releases who are > spending time on that. I have been following this project for a couple > years and the only people who are taking care of releases are Brandon and > Mick. I was helping here and there to at least stage it and I am willing to > continue to do so, but that is basically it. "two and a half" people are > doing releases. For all these years. > > So if you ask for more frequent releases, that is something which is going > to directly affect respective people involved in them. I guess they are > doing it basically out of courtesy and it would be great to see more PMCs > involved in release processes. As of now, it looks like everybody just > assumes that "it will be somehow released" and "releases just happen" but > that is not the case. Releases are not "just happening". There are people > behind them who need to plan when it is going to happen and they need to > find time for that etc. There are a lot of things not visible behind the > scenes and doing releases is a job in itself. > > So if we ask for more frequent releases, it is a good question to ask who > would be actually releasing that. > > On Wed, Jan 22, 2025 at 12:17 PM Dmitry Konstantinov > wrote: > >> Hi all, >> >> I am one of the contributors for the recent perf changes, like: >> https://issues.apache.org/jira/browse/CASSANDRA-20165 >> https://issues.apache.org/jira/browse/CA
Re: What branches should perf fixes be targeting
>> That is ... 6 branches at once. We were there, 3.0, 3.11, 4.0, 4.1, 5.0, trunk. If there was a bug in 3.0, because we were supporting that, we had to put this into 6 branches My idea is not to increase the number of support branches (it is definitely not what I want to, I am more a fan of release-ready trunk-based development with a faster feedback loop, but it is not always applicable). The option was about releasing non-long term support minor versions: like JDK release JDK 9/10 as short term and then JDK11 as long term, then 12/13 as short term and so on. So, in the case of Cassandra for example, we now have 5.0.x as a long term support version with a branch, we can release 5.1/5.2 from trunk (without any new support branches for them) and then 5.3 as a long term again with a bug fix branch. The overhead here is only for the more frequent release (like once per 3 or 6 months), there is no overhead for branches/merges. On Thu, 23 Jan 2025 at 14:31, Štefan Miklošovič wrote: > > > On Thu, Jan 23, 2025 at 3:20 PM Dmitry Konstantinov > wrote: > >> Hi Stefan, >> >> Thank you a lot for the detailed feedback! Few comments: >> >> >> I think this is already the case, more or less. We are not doing perf >> changes in older branches. >> Yes, I understand the idea about stability of older branches, the primary >> issue for me is that if I contribute even a small improvement to trunk - I >> cannot really use it for a long time (except having it in my own >> fork), because there is no release to get it back for me or anybody else.. >> >> >> Maybe it would be better to make the upgrading process as smooth as >> possible so respective businesses are open to upgrade their clusters in a >> more frequent manner. >> About the upgrade process: my personal experience (3.0.x -> 3.11.x -> >> 4.0.x -> 4.1.x), the upgrade in Cassandra is positive (I suppose the >> autotests which test it are really helpful), I have not experienced any >> serious issues with it. I suppose the majority of time when people have an >> issue with upgrades is due to delaying them for too long and staying on >> very old unsupported versions till the last moment. >> >> >> Cassandra is not JDK. We need to fix bugs in older branches we said >> we support >> Regarding the necessity to support the older branches it is the same >> story for JDK: they now support and fix bugs in JDK8, JDK11, JDK17 and JDK >> 21 as LTS versions and JDK23 as the latest release while developing and >> releasing JDK24 now. >> > > That is ... 6 branches at once. We were there, 3.0, 3.11, 4.0, 4.1, 5.0, > trunk. If there was a bug in 3.0, because we were supporting that, we had > to put this into 6 branches. That means 6 builds in CI. Each CI takes a > couple hours ... If there is something wrong or the patch is changed we > need to rebuild. So what looks like "just merge up from 3.0 and that's it" > becomes a multi-day odyssey somebody needs to invest resources into. As we > dropped 3.0 and 3.11 and we took care of 4.0+ that is better but still not > fun when done "at scale". > > >> Another example, Postgres does a major release every year: >> https://www.postgresql.org/support/versioning/ and supports the last 5 >> major versions. >> > > Yeah, but they have most probably way more man-power as well etc ... > > >> >> >> please keep in mind that there are people behind the releases who are >> spending time on that. >> Yes, as I already mentioned, I really thank you to Brandon and Mick for >> doing it! It is hard, exhausting and not the most exciting work to do. >> Please contact me if I can help somehow with it, like checking and fixing >> CI test failures(I've already done it for a while) / doing some scripting/ >> etc. >> I have a hypothesis (maybe I am completely wrong here) that actually the >> low interest in the releasing process is somehow related to having a >> Cassandra fork by many contributors, so there is no big demand for regular >> mainline releases if you have them in a fork.. >> >> Regards, >> Dmitry >> >> >> >> >> >> >> >> >> >> On Thu, 23 Jan 2025 at 12:30, Štefan Miklošovič >> wrote: >> >>> I think the current guidelines are sensible. >>> >>> Going through your suggestions: >>> >>> 1) I think this is already the case, more or less. We are not doing perf >>> changes in older branches. This is what we see in CASSANDRA-19429, a user >>> reported that it is a performance improvement, and most probably he is >>>
Re: [DISCUSS] synchronisation of properties between Config.java and cassandra.yaml
in runtime as it fails on unrecognized property. > > This will be used in practice very rarely as we seldom remove the > properties in Config but if we do and a property is commented out, we > should not ship a dead property name, even commented out. > > On Fri, Jan 24, 2025 at 3:51 PM Paulo Motta wrote: > >> > > If "# my_cool_property: true" is NOT in cassandra.yaml, we might >> indeed add it, also commented out. I think it would be quite easy to check >> against yaml if there is a line starting on "# my_cool_property" or just on >> "my_cool_property". Both cases would satisfy the check. >> >> Makes sense, I think this would be good to have as a lint or test to >> easily catch overlooks during review. >> >> >> On Fri, Jan 24, 2025 at 9:44 AM Štefan Miklošovič >> wrote: >> >>> >>> >>> On Fri, Jan 24, 2025 at 3:27 PM Paulo Motta wrote: >>> >>>> > from time to time I see configuration properties in Config.java and >>>> they are clearly not in cassandra.yaml. Not every property in Config is in >>>> cassandra.yaml. I would like to know if there is some specific reason >>>> behind that. >>>> >>>> I think one of the original reasons was to "hide" advanced configs that >>>> are not meant to be updated, unless in very niche circumstances. However I >>>> think this has been extrapolated to non-advanced settings. >>>> >>>> > Question related to that is if we could not have a build-time check >>>> that all properties in Config have to be in cassandra.yaml and fail the >>>> build if a property in Config does not have its counterpart in yaml. >>>> >>>> Are you saying every configuration property should be commented-out, or >>>> do you think that every Config property should be specified in >>>> cassandra.yaml with their default uncomented ? One issue with that is that >>>> you could cause user confusion if you "reveal" a niche/advanced config that >>>> is not meant to be updated. I think this would be addressed by >>>> the @HiddenInYaml flag you are proposing in a later post. >>>> >>> >>> Yes, then can stay hidden, but we should annotate it with @Hidden or >>> similar. As of now, if that property is not in yaml, we just don't know if >>> it was forgotten to be added or if we have not added it on purpose. >>> >>> They can keep being commented out if they currently are. Imagine a >>> property in Config.java >>> >>> public boolean my_cool_property = true; >>> >>> and then this in cassandra.yaml >>> >>> # my_cool_property: true >>> >>> It is completely ok. >>> >>> If "# my_cool_property: true" is NOT in cassandra.yaml, we might indeed >>> add it, also commented out. I think it would be quite easy to check against >>> yaml if there is a line starting on "# my_cool_property" or just on >>> "my_cool_property". Both cases would satisfy the check. >>> >>> >>> >>>> > There are dozens of properties in Config and I have a strong >>>> suspicion that we missed to publish some to yaml so users do not even know >>>> such a property exists and as of now we do not even know which they are. >>>> >>>> I believe this is a problem. I think most properties should be in >>>> cassandra.yaml, unless they are very advanced or not meant to be updated. >>>> >>>> Another tangential issue is that there are features/settings that don't >>>> even have a Config entry, but are just controlled by JVM properties. >>>> >>>> I think that we should attempt to unify Config and jvm properties under >>>> a predictable structure. For example, if there is a YAML config >>>> enable_user_defined_functions, then there should be a respective JVM flag >>>> -Dcassandra.enable_user_defined_functions, and vice versa. >>>> >>> >>> Yeah, good idea. >>> >>> >>>> >>>> On Fri, Jan 24, 2025 at 9:16 AM Štefan Miklošovič < >>>> smikloso...@apache.org> wrote: >>>> >>>>> Hello, >>>>> >>>>> from time to time I see configuration properties in Config.java and >>>>> they are clearly not in cassandra.yaml. Not every property in Config is in >>>>> cassandra.yaml. I would like to know if there is some specific reason >>>>> behind that. >>>>> >>>>> Question related to that is if we could not have a build-time check >>>>> that all properties in Config have to be in cassandra.yaml and fail the >>>>> build if a property in Config does not have its counterpart in yaml. >>>>> >>>>> There are dozens of properties in Config and I have a strong suspicion >>>>> that we missed to publish some to yaml so users do not even know such a >>>>> property exists and as of now we do not even know which they are. >>>>> >>>> -- Dmitry Konstantinov
Re: [DISCUSS] synchronisation of properties between Config.java and cassandra.yaml
Maybe I missed some patterns but it looks like a pretty good estimation, I
did like 10 random checks manually to verify :-)
I will try to make an ant target with a similar logic (hopefully, during
the weekend)
I will create a ticket to track this activity (to share attachments there
to not overload the thread with such outputs in future).
On Fri, 24 Jan 2025 at 15:37, Štefan Miklošovič
wrote:
> Oh my god, 112? :DD I was thinking it would be less than 10.
>
> Anyway, I think we need to integrate this to some ant target. If you
> expanded on this, that would be great.
>
> On Fri, Jan 24, 2025 at 4:31 PM Dmitry Konstantinov
> wrote:
>
>> A very primitive implementation of the 1st idea below:
>>
>> String configUrl =
>> "file:///Users/dmitry/IdeaProjects/cassandra-trunk/conf/cassandra.yaml";
>> Field[] allFields = Config.class.getFields();
>> List topLevelPropertyNames = new ArrayList<>();
>> for(Field field : allFields)
>> {
>> if (!Modifier.isStatic(field.getModifiers()))
>> {
>> topLevelPropertyNames.add(field.getName());
>> }
>> }
>>
>> URL url = new URL(configUrl);
>> List lines = Files.readAllLines(Paths.get(url.toURI()));
>>
>> int missedCount = 0;
>> for (String propertyName : topLevelPropertyNames)
>> {
>> boolean found = false;
>> for (String line : lines)
>> {
>> if (line.startsWith(propertyName + ":")
>> || line.startsWith("#" + propertyName + ":")
>> || line.startsWith("# " + propertyName + ":")) {
>> found = true;
>> break;
>> }
>> }
>> if (!found)
>> {
>> missedCount++;
>> System.out.println(propertyName);
>> }
>> }
>> System.out.println("Total missed:" + missedCount);
>>
>>
>> It prints the following config property names which are defined in
>> Config.java but not present as "property" or "# property " in a file:
>>
>> permissions_cache_max_entries
>> roles_cache_max_entries
>> credentials_cache_max_entries
>> auto_bootstrap
>> force_new_prepared_statement_behaviour
>> use_deterministic_table_id
>> repair_request_timeout
>> stream_transfer_task_timeout
>> cms_await_timeout
>> cms_default_max_retries
>> cms_default_retry_backoff
>> epoch_aware_debounce_inflight_tracker_max_size
>> metadata_snapshot_frequency
>> available_processors
>> repair_session_max_tree_depth
>> use_offheap_merkle_trees
>> internode_max_message_size
>> native_transport_max_message_size
>> native_transport_max_request_data_in_flight_per_ip
>> native_transport_max_request_data_in_flight
>> native_transport_receive_queue_capacity
>> min_free_space_per_drive
>> max_space_usable_for_compactions_in_percentage
>> reject_repair_compaction_threshold
>> concurrent_index_builders
>> max_streaming_retries
>> commitlog_max_compression_buffers_in_pool
>> max_mutation_size
>> dynamic_snitch
>> failure_detector
>> use_creation_time_for_hint_ttl
>> key_cache_migrate_during_compaction
>> key_cache_invalidate_after_sstable_deletion
>> paxos_cache_size
>> file_cache_round_up
>> disk_optimization_estimate_percentile
>> disk_optimization_page_cross_chance
>> purgeable_tobmstones_metric_granularity
>> windows_timer_interval
>> otc_coalescing_strategy
>> otc_coalescing_window_us
>> otc_coalescing_enough_coalesced_messages
>> otc_backlog_expiration_interval_ms
>> scripted_user_defined_functions_enabled
>> user_defined_functions_threads_enabled
>> allow_insecure_udfs
>> allow_extra_insecure_udfs
>> user_defined_functions_warn_timeout
>> user_defined_functions_fail_timeout
>> user_function_timeout_policy
>> back_pressure_enabled
>> back_pressure_strategy
>> repair_command_pool_full_strategy
>> repair_command_pool_size
>> block_for_peers_timeout_in_secs
>> block_for_peers_in_remote_dcs
>> skip_stream_disk_space_check
>> snapshot_on_repaired_data_mismatch
>> validation_preview_purge_head_start
>> initial_range_tombstone_list_allocation_size
>> range_tombstone_list_growth_factor
>> snapshot_on_duplicate_row_detection
>> check_for_duplicate_rows_during_reads
>> check_for_duplicate_rows_during_compaction
>> autocompaction_on_startup_enabled
>> auto_optimise_inc_repair_streams
>> auto_optimise_full_repair_streams
>> auto_optimise_preview_repair_streams
Re: [DISCUSS] synchronisation of properties between Config.java and cassandra.yaml
https://issues.apache.org/jira/browse/CASSANDRA-20249
On Fri, 24 Jan 2025 at 15:40, Dmitry Konstantinov
wrote:
> Maybe I missed some patterns but it looks like a pretty good estimation, I
> did like 10 random checks manually to verify :-)
> I will try to make an ant target with a similar logic (hopefully, during
> the weekend)
> I will create a ticket to track this activity (to share attachments there
> to not overload the thread with such outputs in future).
>
> On Fri, 24 Jan 2025 at 15:37, Štefan Miklošovič
> wrote:
>
>> Oh my god, 112? :DD I was thinking it would be less than 10.
>>
>> Anyway, I think we need to integrate this to some ant target. If you
>> expanded on this, that would be great.
>>
>> On Fri, Jan 24, 2025 at 4:31 PM Dmitry Konstantinov
>> wrote:
>>
>>> A very primitive implementation of the 1st idea below:
>>>
>>> String configUrl =
>>> "file:///Users/dmitry/IdeaProjects/cassandra-trunk/conf/cassandra.yaml";
>>> Field[] allFields = Config.class.getFields();
>>> List topLevelPropertyNames = new ArrayList<>();
>>> for(Field field : allFields)
>>> {
>>> if (!Modifier.isStatic(field.getModifiers()))
>>> {
>>> topLevelPropertyNames.add(field.getName());
>>> }
>>> }
>>>
>>> URL url = new URL(configUrl);
>>> List lines = Files.readAllLines(Paths.get(url.toURI()));
>>>
>>> int missedCount = 0;
>>> for (String propertyName : topLevelPropertyNames)
>>> {
>>> boolean found = false;
>>> for (String line : lines)
>>> {
>>> if (line.startsWith(propertyName + ":")
>>> || line.startsWith("#" + propertyName + ":")
>>> || line.startsWith("# " + propertyName + ":")) {
>>> found = true;
>>> break;
>>> }
>>> }
>>> if (!found)
>>> {
>>> missedCount++;
>>> System.out.println(propertyName);
>>> }
>>> }
>>> System.out.println("Total missed:" + missedCount);
>>>
>>>
>>> It prints the following config property names which are defined in
>>> Config.java but not present as "property" or "# property " in a file:
>>>
>>> permissions_cache_max_entries
>>> roles_cache_max_entries
>>> credentials_cache_max_entries
>>> auto_bootstrap
>>> force_new_prepared_statement_behaviour
>>> use_deterministic_table_id
>>> repair_request_timeout
>>> stream_transfer_task_timeout
>>> cms_await_timeout
>>> cms_default_max_retries
>>> cms_default_retry_backoff
>>> epoch_aware_debounce_inflight_tracker_max_size
>>> metadata_snapshot_frequency
>>> available_processors
>>> repair_session_max_tree_depth
>>> use_offheap_merkle_trees
>>> internode_max_message_size
>>> native_transport_max_message_size
>>> native_transport_max_request_data_in_flight_per_ip
>>> native_transport_max_request_data_in_flight
>>> native_transport_receive_queue_capacity
>>> min_free_space_per_drive
>>> max_space_usable_for_compactions_in_percentage
>>> reject_repair_compaction_threshold
>>> concurrent_index_builders
>>> max_streaming_retries
>>> commitlog_max_compression_buffers_in_pool
>>> max_mutation_size
>>> dynamic_snitch
>>> failure_detector
>>> use_creation_time_for_hint_ttl
>>> key_cache_migrate_during_compaction
>>> key_cache_invalidate_after_sstable_deletion
>>> paxos_cache_size
>>> file_cache_round_up
>>> disk_optimization_estimate_percentile
>>> disk_optimization_page_cross_chance
>>> purgeable_tobmstones_metric_granularity
>>> windows_timer_interval
>>> otc_coalescing_strategy
>>> otc_coalescing_window_us
>>> otc_coalescing_enough_coalesced_messages
>>> otc_backlog_expiration_interval_ms
>>> scripted_user_defined_functions_enabled
>>> user_defined_functions_threads_enabled
>>> allow_insecure_udfs
>>> allow_extra_insecure_udfs
>>> user_defined_functions_warn_timeout
>>> user_defined_functions_fail_timeout
>>> user_function_timeout_policy
>>> back_pressure_enabled
>>> back_pressure_strategy
>>> repair_command_pool_full_strategy
>>> repair_command_pool_size
>>> block_for_
Re: [DISCUSS] synchronisation of properties between Config.java and cassandra.yaml
Hi, I've just thought about the same as Stefan - can we automate this consistency check.. (the topic is actually triggered by our recent review). I definitely saw some other existing options which are available in Config but not mentioned in cassandra.yaml. The case with intentionally hidden options can be covered using some annotation like @Hidden for such properties in Config.java It will allow you to declare the intent to not expose this option clearly. The main complexity from technical point of view I see is that we declare some of the properties only in comments in cassandra.yaml and it can be non-trivial to parse them. update: Stefan has suggested the annotation as well while I was writing the mail :-) On Fri, 24 Jan 2025 at 14:26, Paulo Motta wrote: > > from time to time I see configuration properties in Config.java and they > are clearly not in cassandra.yaml. Not every property in Config is in > cassandra.yaml. I would like to know if there is some specific reason > behind that. > > I think one of the original reasons was to "hide" advanced configs that > are not meant to be updated, unless in very niche circumstances. However I > think this has been extrapolated to non-advanced settings. > > > Question related to that is if we could not have a build-time check that > all properties in Config have to be in cassandra.yaml and fail the build if > a property in Config does not have its counterpart in yaml. > > Are you saying every configuration property should be commented-out, or do > you think that every Config property should be specified in cassandra.yaml > with their default uncomented ? One issue with that is that you could cause > user confusion if you "reveal" a niche/advanced config that is not meant to > be updated. I think this would be addressed by the @HiddenInYaml flag you > are proposing in a later post. > > > There are dozens of properties in Config and I have a strong suspicion > that we missed to publish some to yaml so users do not even know such a > property exists and as of now we do not even know which they are. > > I believe this is a problem. I think most properties should be in > cassandra.yaml, unless they are very advanced or not meant to be updated. > > Another tangential issue is that there are features/settings that don't > even have a Config entry, but are just controlled by JVM properties. > > I think that we should attempt to unify Config and jvm properties under a > predictable structure. For example, if there is a YAML config > enable_user_defined_functions, then there should be a respective JVM flag > -Dcassandra.enable_user_defined_functions, and vice versa. > > On Fri, Jan 24, 2025 at 9:16 AM Štefan Miklošovič > wrote: > >> Hello, >> >> from time to time I see configuration properties in Config.java and they >> are clearly not in cassandra.yaml. Not every property in Config is in >> cassandra.yaml. I would like to know if there is some specific reason >> behind that. >> >> Question related to that is if we could not have a build-time check that >> all properties in Config have to be in cassandra.yaml and fail the build if >> a property in Config does not have its counterpart in yaml. >> >> There are dozens of properties in Config and I have a strong suspicion >> that we missed to publish some to yaml so users do not even know such a >> property exists and as of now we do not even know which they are. >> > -- Dmitry Konstantinov
Re: [DISCUSS] synchronisation of properties between Config.java and cassandra.yaml
> I think we need to integrate this to some ant target. If you expanded on
this, that would be great.
A draft version of the ant target (as of now it is configured as non-failed
and not attached to the usual build process):
https://github.com/apache/cassandra/pull/3830/files
On Fri, 24 Jan 2025 at 20:24, Štefan Miklošovič
wrote:
> How are we going to document what each of 112 missing properties is doing
> and / or exclude them from cassandra.yaml? There are a lot of properties
> which just don't ring a bell exactly what they are for. I think we should
> create a basic table and document what each is for and what is the decision
> about adding that property to yaml or not.
>
> On Fri, Jan 24, 2025 at 4:31 PM Dmitry Konstantinov
> wrote:
>
>> A very primitive implementation of the 1st idea below:
>>
>> String configUrl =
>> "file:///Users/dmitry/IdeaProjects/cassandra-trunk/conf/cassandra.yaml";
>> Field[] allFields = Config.class.getFields();
>> List topLevelPropertyNames = new ArrayList<>();
>> for(Field field : allFields)
>> {
>> if (!Modifier.isStatic(field.getModifiers()))
>> {
>> topLevelPropertyNames.add(field.getName());
>> }
>> }
>>
>> URL url = new URL(configUrl);
>> List lines = Files.readAllLines(Paths.get(url.toURI()));
>>
>> int missedCount = 0;
>> for (String propertyName : topLevelPropertyNames)
>> {
>> boolean found = false;
>> for (String line : lines)
>> {
>> if (line.startsWith(propertyName + ":")
>> || line.startsWith("#" + propertyName + ":")
>> || line.startsWith("# " + propertyName + ":")) {
>> found = true;
>> break;
>> }
>> }
>> if (!found)
>> {
>> missedCount++;
>> System.out.println(propertyName);
>> }
>> }
>> System.out.println("Total missed:" + missedCount);
>>
>>
>> It prints the following config property names which are defined in
>> Config.java but not present as "property" or "# property " in a file:
>>
>> permissions_cache_max_entries
>> roles_cache_max_entries
>> credentials_cache_max_entries
>> auto_bootstrap
>> force_new_prepared_statement_behaviour
>> use_deterministic_table_id
>> repair_request_timeout
>> stream_transfer_task_timeout
>> cms_await_timeout
>> cms_default_max_retries
>> cms_default_retry_backoff
>> epoch_aware_debounce_inflight_tracker_max_size
>> metadata_snapshot_frequency
>> available_processors
>> repair_session_max_tree_depth
>> use_offheap_merkle_trees
>> internode_max_message_size
>> native_transport_max_message_size
>> native_transport_max_request_data_in_flight_per_ip
>> native_transport_max_request_data_in_flight
>> native_transport_receive_queue_capacity
>> min_free_space_per_drive
>> max_space_usable_for_compactions_in_percentage
>> reject_repair_compaction_threshold
>> concurrent_index_builders
>> max_streaming_retries
>> commitlog_max_compression_buffers_in_pool
>> max_mutation_size
>> dynamic_snitch
>> failure_detector
>> use_creation_time_for_hint_ttl
>> key_cache_migrate_during_compaction
>> key_cache_invalidate_after_sstable_deletion
>> paxos_cache_size
>> file_cache_round_up
>> disk_optimization_estimate_percentile
>> disk_optimization_page_cross_chance
>> purgeable_tobmstones_metric_granularity
>> windows_timer_interval
>> otc_coalescing_strategy
>> otc_coalescing_window_us
>> otc_coalescing_enough_coalesced_messages
>> otc_backlog_expiration_interval_ms
>> scripted_user_defined_functions_enabled
>> user_defined_functions_threads_enabled
>> allow_insecure_udfs
>> allow_extra_insecure_udfs
>> user_defined_functions_warn_timeout
>> user_defined_functions_fail_timeout
>> user_function_timeout_policy
>> back_pressure_enabled
>> back_pressure_strategy
>> repair_command_pool_full_strategy
>> repair_command_pool_size
>> block_for_peers_timeout_in_secs
>> block_for_peers_in_remote_dcs
>> skip_stream_disk_space_check
>> snapshot_on_repaired_data_mismatch
>> validation_preview_purge_head_start
>> initial_range_tombstone_list_allocation_size
>> range_tombstone_list_growth_factor
>> snapshot_on_duplicate_row_detection
>> check_for_duplicate_rows_during_reads
>> check_for_duplicate_rows_during_compaction
>> autocompaction_on_startup
Re: Patch Available vs Needs Committer
> Jira actually does that and creates a diagram, Yes, I am aware about this functionality, unfortunately it does not show the most interesting part: conditions/expectations for transitions I have combined the following summary based on comments from Benedict, the docs and a discussion with Stefan: *The flow when author is a committer:* 1) In Progress -- an author has attached a patch as MR or patch file --> Needs Committer (or Patch Available) 2) Needs Committer/Patch Available -- a review is started by other committers/contributors --> Review in Progress 3) Review in Progress -- some reviewers added comments to address --> Changes Suggested 4) Changes Suggested -- the author addressed review comments and update the patch --> Needs Committer/Patch Available 5) Review in Progress -- review comments are addressed; at least one another committer does a full review and set +1 --> Ready For Commit 6) Ready For Commit -- author or another committer merges the patch to release branches, update the fix version and commit link --> Resolved *The flow when author is a contributor (non-committer):* 1) In Progress -- an author has attached a patch as MR or patch file --> Patch Available 2) Patch Available -- a review is started by other committers/contributors --> Review in Progress 3) Review in Progress -- some reviewers added comments to address --> Changes Suggested 4) Changes Suggested -- the author addressed review comments and update the patch --> Patch Available 5) Review in Progress -- review comments are addressed; at least one committer does a full review and set +1 --> Needs Committer 6) Needs Committer -- another committer set +1, so in total at least two +1 from committers --> Ready For Commit 6) Ready For Commit -- some committer merges the patch to release branches, update the fix version and commit link --> Resolved *Notes:* 1) there is an implicit +1 from the case when author = committer, so in total - it is always at least two +1 from committers 2) contributors (non-committers) can participate in review but cannot set binding +1s 3) actual merge can be done by any committer, not necessarily the author or a reviewer 4) Interim transitions between Patch Available and Resolved can be skipped by a committer as an author if all quality gates are still passed (+1 is received, CI results posted, fix version and commit link are updated) Regards, Dmitry On Wed, 29 Jan 2025 at 16:32, Brandon Williams wrote: > On Wed, Jan 29, 2025 at 10:26 AM Dmitry Konstantinov > wrote: > > I want to draw it explicitly (as a diagram or as a table) and share to > check my understanding, then after a discussion it can be integrated in the > contributor docs. > > I hope it makes sense.. > > Jira actually does that and creates a diagram, if you look at the > pinned items in #cassandra-dev on slack there is one that says "The > great jira workflow map, 9/21" and I have confirmed it's still the > same today. > > Kind Regards, > Brandon > -- Dmitry Konstantinov
Re: Patch Available vs Needs Committer
Hi all, I have used Cassandra for a long time and had my fork to make changes but recently I have started to contribute more actively to the upstream. As a quite a new contributor I am a bit confused with a lack of clarity for ticket transition rules. I understand that for the majority there is no real need for it - people just have it learned from practice but probably we can use me as a tester for the process description to make it more clear and explicit. I read the following docs: https://cassandra.apache.org/_/development/index.html https://cassandra.apache.org/_/development/patches.html but still I do not have a very clear picture of a ticket lifecycle (especially about the part with reviewing/approvals). For me as an engineer it is a finite state machine with conditions on transitions between states. I want to draw it explicitly (as a diagram or as a table) and share to check my understanding, then after a discussion it can be integrated in the contributor docs. I hope it makes sense.. Regards, Dmitry On Mon, 27 Jan 2025 at 13:55, Brandon Williams wrote: > Hello, > > This is just a reminder of the difference between these states. > "Needs committer" is for when there is already one reviewer, and a > second is needed (this state used to be called "Needs 2nd reviewer".) > "Patch Available" is the state used when you need an initial review. > If in doubt, use "Patch Available" please, and thanks! > > https://lists.apache.org/thread/g4mvx41jdvn700719po0nxhz7n1ftpnc > > Kind Regards, > Brandon > -- Dmitry Konstantinov
Re: [DISCUSS] Replacement of SSTable's partition cardinality implementation from stream-lib to Apache Datasketches
t; from one HLL to another. >> >> The proposal to reduce allocations appears to be orthogonal to this >> library, so let’s separate out that discussion? If there’s evidence this >> library alone improves the memory profile let’s discuss that. >> >> >> On 2 Jan 2025, at 15:26, Chris Lohfink wrote: >> >> >> I think switching to datasketches is a good idea first off simply because >> of the lack of maintenance and improvements from clearspring. I am however, >> am not sold that it will actually improve anything significantly. Caches >> might help on small cases, but those small cases probably are not actually >> impacted. In the large cases the caches cost more in complexity, memory, >> and ultimately wont matter when theres 50k sstables and the cache holds 1k >> so everythings hitting disk anyway. >> >> The 5% is missing some relevant information like what the allocation rate >> was, how many tables there are etc. On an idle system thats meaningless, if >> there were 5gb/s allocations of reads/writes happening at the time thats >> huge. >> >> On Thu, Jan 2, 2025 at 8:42 AM Štefan Miklošovič >> wrote: >> >>> Interesting, thanks for this. Well ... 5% here, 5% there ... it >>> compounds. I think it is worth trying to do something with this. Would be >>> great if you were part of this effort! >>> >>> On Thu, Jan 2, 2025 at 3:38 PM Dmitry Konstantinov >>> wrote: >>> >>>> I have seen this place in async profiler memory allocation profile on >>>> one of production environments some time ago, it was visible but not >>>> dramatic, about 5% of allocations: >>>> >>>> >>>> The amount of overhead also depends on a metric collection frequency >>>> (in my case it was once per 60 seconds) >>>> >>>> Regards, >>>> Dmitry >>>> >>>> On Thu, 2 Jan 2025 at 14:21, Štefan Miklošovič >>>> wrote: >>>> >>>>> Indeed, I plan to measure it and compare, maybe some bench test would >>>>> be cool to add .. >>>>> >>>>> I strongly suspect that the primary reason for the slowness (if it is >>>>> verified to be true) is us going to the disk every time and reading stats >>>>> for every SSTable all over again. >>>>> >>>>> While datasketches say that it is way faster to update (1), we are >>>>> living in a realm of nanoseconds here and I don't think that itself would >>>>> make any meaningful difference when merging one hyperloglog with another >>>>> as >>>>> part of partition rows estimation computation. The only place we are >>>>> updating is SortableTableWriter#endParition which calls >>>>> metadatacollector.addKey(key.getKey()) which eventually updates the >>>>> estimator via cardinality#offeredHashed. >>>>> >>>>> In other words, I think that going to the disk and reading it >>>>> repeatedly is disproportionally more IO / time intensive than switching >>>>> the >>>>> hyperloglog implementation. >>>>> >>>>> However, I consider the replacement of the library still important. I >>>>> feel uneasy about staying with an abandoned library where there is clearly >>>>> a well-maintained replacement. >>>>> >>>>> What we could do is to cache all cardinality estimators and just merge >>>>> it all when asked upon metric resolution. That is different from going to >>>>> disk to deserialize StatsComponent's all over again. >>>>> >>>>> Then on SSTable removal, we would remove that from cache too. I think >>>>> there is some kind of an observer when SSTable is removed ... >>>>> >>>>> However, I am not sure I can just hold it all in the memory, it works >>>>> for laptop testing but if we have thousands of SSTables with non-trivial >>>>> number of rows things start to get interesting. >>>>> >>>>> (1) https://datasketches.apache.org/docs/HLL/Hll_vs_CS_Hllpp.html - >>>>> section HllSketch vs. HyperLogLogPlus Update Speed Behavior >>>>> >>>>> On Thu, Jan 2, 2025 at 2:46 PM Jon Haddad >>>>> wrote: >>>>> >>>>>> Sounds interesting. I took a look at the issue but I'm not seeing >>>>>> any data to back up "expen
Re: [DISCUSS] CEP-45: Mutation Tracking
ince we >> lack a consistency mechanism like MVCC while paginating, it's possible to >> do the following: >> >> thread A: reads a partition P with 10K rows, starts by reading the first >> page >> thread B: another thread writes a batch to 2 rows in partition P, one on >> page 1, another on page 2 >> thread A: reads the second page of P which has the mutation. >> >> I've worked with users who have been surprised by this behavior, because >> pagination happens transparently. >> >> So even without repair mucking things up, we're unable to fulfill this >> promise except under the specific, ideal circumstance of querying a >> partition with only 1 page. >> >> Jon >> >> >> >> >> >> On Wed, Jan 8, 2025 at 11:21 AM Blake Eggleston >> wrote: >> >>> Hello dev@, >>> >>> We'd like to propose CEP-45: Mutation Tracking for adoption by the >>> community. CEP-45 proposes adding a replication mechanism to track and >>> reconcile individual mutations, as well as processes to actively reconcile >>> missing mutations. >>> >>> For keyspaces with mutation tracking enabled, the immediate benefits of >>> this CEP are: >>> * reduced replication lag with a continuous background reconciliation >>> process >>> * eliminate the disk load caused by repair merkle tree calculation >>> * eliminate repair overstreaming >>> * reduce disk load of reads on cluster to close to 1/CL >>> * fix longstanding mutation atomicity issues caused by read repair and >>> short read protection >>> >>> Additionally, although it's outside the scope of this CEP, mutation >>> tracking would enable: >>> * completion of witness replicas / transient replication, making the >>> feature usable for all workloads >>> * lightweight witness only datacenters >>> >>> The CEP is linked here: >>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-45%3A+Mutation+Tracking, >>> but please keep the discussion on the dev list. >>> >>> Thanks! >>> >>> Blake Eggleston >>> >> >> -- Dmitry Konstantinov
Re: [DISCUSS] Replacement of SSTable's partition cardinality implementation from stream-lib to Apache Datasketches
Let me clarify my comment regarding allocation. I am not saying that switching to another implementation will make it better and we need to do it right now :-), any such switch is a subject for pros/cons analysis (and memory allocation I think should be one of criteria). What I wanted to say: this place has attracted my attention as potentially suspicious from another perspective, so it makes sense to analyse it in more detail, not more as of now. Regarding allocation details. The DB host had the following stats at that time: 5K/sec local reads, 3K/sec local writes, about 800 live SSTables, the profile was collected with duration = 5 minutes. I do not have an allocation rate info for that time period. On Thu, 2 Jan 2025 at 16:02, Benedict wrote: > I am strongly opposed to updating libraries simply for the sake of it. > Something like HLL does not need much ongoing maintenance if it works. > We’re simply asking for extra work and bugs by switching, and some risk > without understanding the quality control for the new library project’s > releases. > > That said, I was not very impressed with the clear spring library when I > looked at it, so I would be open to a stronger argument about data sketches > being superior otherwise in a way that matters to us. > > If we are to replace the library, we should at the very least do proper > due diligence by reviewing the new library’s implementation(s) ourselves. > We cannot simply assume the new library behaves well for our use cases, or > is well maintained. > > We should also not use the fallback intersection method, as this would > represent a regression to compaction on upgrade. We should really convert > from one HLL to another. > > The proposal to reduce allocations appears to be orthogonal to this > library, so let’s separate out that discussion? If there’s evidence this > library alone improves the memory profile let’s discuss that. > > > On 2 Jan 2025, at 15:26, Chris Lohfink wrote: > > > I think switching to datasketches is a good idea first off simply because > of the lack of maintenance and improvements from clearspring. I am however, > am not sold that it will actually improve anything significantly. Caches > might help on small cases, but those small cases probably are not actually > impacted. In the large cases the caches cost more in complexity, memory, > and ultimately wont matter when theres 50k sstables and the cache holds 1k > so everythings hitting disk anyway. > > The 5% is missing some relevant information like what the allocation rate > was, how many tables there are etc. On an idle system thats meaningless, if > there were 5gb/s allocations of reads/writes happening at the time thats > huge. > > On Thu, Jan 2, 2025 at 8:42 AM Štefan Miklošovič > wrote: > >> Interesting, thanks for this. Well ... 5% here, 5% there ... it >> compounds. I think it is worth trying to do something with this. Would be >> great if you were part of this effort! >> >> On Thu, Jan 2, 2025 at 3:38 PM Dmitry Konstantinov >> wrote: >> >>> I have seen this place in async profiler memory allocation profile on >>> one of production environments some time ago, it was visible but not >>> dramatic, about 5% of allocations: >>> >>> >>> The amount of overhead also depends on a metric collection frequency (in >>> my case it was once per 60 seconds) >>> >>> Regards, >>> Dmitry >>> >>> On Thu, 2 Jan 2025 at 14:21, Štefan Miklošovič >>> wrote: >>> >>>> Indeed, I plan to measure it and compare, maybe some bench test would >>>> be cool to add .. >>>> >>>> I strongly suspect that the primary reason for the slowness (if it is >>>> verified to be true) is us going to the disk every time and reading stats >>>> for every SSTable all over again. >>>> >>>> While datasketches say that it is way faster to update (1), we are >>>> living in a realm of nanoseconds here and I don't think that itself would >>>> make any meaningful difference when merging one hyperloglog with another as >>>> part of partition rows estimation computation. The only place we are >>>> updating is SortableTableWriter#endParition which calls >>>> metadatacollector.addKey(key.getKey()) which eventually updates the >>>> estimator via cardinality#offeredHashed. >>>> >>>> In other words, I think that going to the disk and reading it >>>> repeatedly is disproportionally more IO / time intensive than switching the >>>> hyperloglog implementation. >>>> >>>> However, I consider the re
Re: Welcome Jeremiah Jordan to the PMC
Congratulations, Jeremiah! On Fri, 14 Feb 2025 at 15:32, Enrico Olivelli wrote: > Congratulations ! > > Enrico > > Il giorno ven 14 feb 2025 alle ore 16:26 Jacek Lewandowski < > lewandowski.ja...@gmail.com> ha scritto: > >> Congratulations!!! >> >> On Fri, Feb 14, 2025, 16:17 Jeremy Hanna >> wrote: >> >>> Congratulations Jeremiah - well deserved. >>> >>> On Feb 14, 2025, at 9:11 AM, Ekaterina Dimitrova >>> wrote: >>> >>> Congrats!! Well deserved! Thank you for all you do and I really >>> appreciate how much you always help everyone, sharing your broad knowledge >>> and expertise! >>> >>> Cheers >>> >>> On Fri, 14 Feb 2025 at 9:36, Brandon Williams wrote: >>> >>>> Congratulations Jeremiah! >>>> >>>> Kind Regards, >>>> Brandon >>>> >>>> On Fri, Feb 14, 2025 at 8:32 AM Benedict Elliott Smith >>>> wrote: >>>> > >>>> > The PMC is happy to announce that Jeremiah Jordan has joined its >>>> membership. >>>> > >>>> > Jeremiah has been a member of this community for almost 15 years. I >>>> hope you will join me in welcoming him to the committee. >>>> >>> >>> -- Dmitry Konstantinov
Re: Huge NetApp donation of hardware for ci-cassandra
Thank you a lot to NetApp and to all who helped to make this happen! On Wed, 19 Mar 2025 at 18:07, Ekaterina Dimitrova wrote: > Thanks a lot! Truly appreciate it! > > On Wed, 19 Mar 2025 at 18:27, Paulo Motta wrote: > >> As an exclusive user of ci-cassandra.a.o I'm very grateful for this >> donation. 🙏 >> >> I never got circle-ci to work on my machine 🥲😉 >> >> Thanks all for making this happen! >> >> On Wed, 19 Mar 2025 at 12:51 Mick Semb Wever wrote: >> >>> >>> Under a ASF targeted sponsorship, NetApp (Instaclustr) has been very >>> generous with the community and donated ten beefy (AMD EPYC 9454P Genoa >>> 48-Core, 256G ram) servers to be used with our ci-cassandra.apache.org >>> infrastructure. >>> >>> On each server we fit 6 jenkins executors, increasing our >>> ci-cassandra.a.o executor count by 42 ! >>> (60 new, minus 18 old executors from Instaclustr now removed). >>> >>> This raises our executor count from 98 to 140, and means NetApp's >>> donation is currently running 30% of the project's CI resources ! >>> >>> This is a big deal for the project, adding both stability and improved >>> throughput of CI for the community. >>> https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md >>> >>> >>> A very big thank you to NetApp, and to all our contributors employed >>> there to help make this happen. >>> >> -- Dmitry Konstantinov
Re: [VOTE] Simplifying our release versioning process
+1 On Tue, 22 Apr 2025 at 16:37, Caleb Rackliffe wrote: > +1 > > On Tue, Apr 22, 2025 at 8:37 AM Ekaterina Dimitrova > wrote: > >> +1 >> >> I also remember we agreed on Discuss thread for removing anything plus >> preference for backward compatibility wherever it is possible. >> >> On Tue, 22 Apr 2025 at 7:00, Sam Tunnicliffe wrote: >> >>> +1 >>> >>> > On 17 Apr 2025, at 16:58, Josh McKenzie wrote: >>> > >>> > [DISCUSS] thread: >>> https://lists.apache.org/thread/jy6vodbkh64plhdfwqz3l3364gsmh2lq >>> > >>> > The proposed new versioning mechanism: >>> > • We no longer use semver .MINOR >>> > • Online upgrades are supported for all GA supported releases at >>> time of new .MAJOR >>> > • T-1 releases are guaranteed API compatible for non-deprecated >>> features >>> > • We use a deprecate-then-remove strategy for API breaking changes >>> (deprecate in release N, then remove in N+1) >>> > This would translate into the following for our upcoming releases >>> (assuming 3 supported majors at all times): >>> > • 6.0: 5.0, 4.1, 4.0 online upgrades are supported (grandfather >>> window). We drop support for 4.0. API compatibility is guaranteed w/5.0 >>> > • 7.0: 6.0, 5.0, 4.1 online upgrades are supported (grandfather >>> window). We drop support for 4.1. API compatibility is guaranteed w/6.0 >>> > • 8.0: 7.0, 6.0, 5.0 online upgrades are supported (fully on new >>> paradigm). We drop support for 5.0. API compatibility guaranteed w/7.0 >>> > David asked the question: >>> >> >>> >> >>> >> Does this imply that each release is allowed to make breaking changes >>> (assuming they followed the “correct” deprecation process)? My first >>> instinct is to not like this >>> > >>> > Each release would be allowed to make breaking changes but only for >>> features that have already been deprecated for one major release cycle. >>> > >>> > This is a process change so as per our governance: >>> https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+Project+Governance, >>> it'll require a super majority of 50% of the roll called PMC in favor. >>> Current roll call is 21 so we need 11 pmc members to participate, 8 of >>> which are in favor of the change. >>> > >>> > I'll plan to leave the vote open until we hit enough participation to >>> pass or fail it up to probably a couple weeks. >>> >>> >>> -- Dmitry Konstantinov
Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)
+1 for the idea to re-implement this logic using python or some java libraries/ant tasks. The fewer languages we have in our build system the easier it is to support and manage at an infrastructure level. On Tue, 29 Apr 2025 at 12:30, Maxim Muzafarov wrote: > Hey, > > I've prepared a python script that generates the same docs (no go > dependency). I use the jinja2 dependency, not sure if it's optimal > because I had to google how to use it though (also not sure if it has > to be run in docker). > I haven't tested the generated files with the website, but I've > compared the results with the same files in the trunk, and they look > similar (almost). > > > https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python > > On Tue, 29 Apr 2025 at 10:10, Benedict wrote: > > > > We should never download and install software via adhoc scripts without > user consent. Was this ever discussed on this mailing list? If not, it’s a > clear breach of policy (introducing a new dependency) and a severe one in > my opinion, as it seems to introduce a new supply chain attack vector for > all developers of Cassandra. > > > > > > > > On 29 Apr 2025, at 08:17, Mick Semb Wever wrote: > > > > > > > > . > > > > > >> > >> But that doesn’t seem to be the case here, the script checks for arm vs > amd64, Linux vs Mac, and then fetches and untars the go distro into tmp. > There is no verification of the download. The only check is if curl > returned non 0. > > > > > > > > Thanks for catching this, the sha256 check should always have been in > place. Adding this is just a one-liner, so that alone shouldn't force the > decision. > > > > > > > >> It looks to be very simple to install Go > > > > > > > > It takes a bit to ensure all build and CI systems are updated, and we > never catch everything (esp what's downstream). > > > > > > While it's "simple", multiplied by everyone (and every system) it adds > up to be a significant time demand. > > > > Again, this too shouldn't be forcing the decision either way on what we > want to do. > > > > > > > -- Dmitry Konstantinov
Re: Welcome Jaydeepkumar Chovatia as Cassandra committer
Congrats Jaydeep! On Wed, 30 Apr 2025 at 13:22, Brandon Williams wrote: > Congrats! > > Kind Regards, > Brandon > > On Wed, Apr 30, 2025 at 6:43 AM Josh McKenzie > wrote: > > > > Hey there Cassandra Devs! > > > > The Apache Cassandra PMC is very happy to announce that Jaydeep Chovatia > has > > accepted the invitation to become a committer! > > > > Jaydeep has been busy on Cassandra for a good while now, most recently > spearheading the contribution of automated repair scheduling via CEP-37 and > CASSANDRA-19918. > > > > Please join us in congratulating and welcoming Jaydeep. > > > > Josh McKenzie on behalf of the Apache Cassandra PMC > -- Dmitry Konstantinov
Re: [DISCUSS] CEP-46 Finish Transient Replication/Witnesses
ster replicated at RF=3 in each DC as an example. In > this topology, Cassandra stores 9× copies of the database forever - huge > storage amplification. Witnesses allow users to maintain a voting quorum of > 9 members (3× per DC); but reduce the durable replicas to 2× per DC – e.g., > two durable replicas and one witness. This maintains the availability > properties of an RF=3×3 topology while reducing storage costs by 33%, going > from 9× copies to 6×. > > > > The role of a witness is to "witness" a write and persist it until it > has been reconciled among all durable replicas; and to respond to read > requests for witnessed writes awaiting reconciliation. Note that witnesses > don't introduce a dedicated role for a node – whether a node is a durable > replica or witness for a token just depends on its position in the ring. > > > > This CEP builds on CEP-45: Mutation Tracking to establish the safety > property of the witness: guaranteeing that writes have been persisted to > all durable replicas before becoming purgeable. CEP-45's journal and > reconciliation design provide a great mechanism to ensure this while > avoiding the write amplification of incremental repair and anticompaction. > > > > Take a look at the CEP if you're interested - happy to answer questions > and discuss further: > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-45%3A+Mutation+Tracking > > > > – Scott > > > > [1] https://cloud.google.com/spanner/docs/replication > > [2] https://www.usenix.org/system/files/atc22-elhemali.pdf > > > >> On Apr 25, 2025, at 8:21 AM, Ariel Weisberg wrote: > >> > >> Hi all, > >> > >> The CEP is available here: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=353601959 > >> > >> We would like to propose CEP-46: Finish Transient Replication/Witnesses > for adoption by the community. CEP-46 would rename transient replication to > witnesses and leverage mutation tracking to implement witnesses as CEP-45 > Mutation Tracking based Log Replicas as a replacement for incremental > repair based witnesses. > >> > >> For those not familiar with transient replication it would have the > keyspace replication settings declare some replicas as transient and when > incremental repair runs the transient replicas would delete data instead of > moving it into the repaired set. > >> > >> With log replicas nodes only materialize mutations in their local LSM > for ranges where they are full replicas and not witnesses. For witness > ranges a node will write mutations to their local mutation tracking log and > participate in background and read time reconciliation. This saves the > compaction overhead of IR based witnesses which have to materialize and > perform compaction on all mutations even those being applied to witness > ranges. > >> > >> This would address one of the biggest issues with witnesses which is > the lack of monotonic reads. Implementation complexity wise this would > actually delete code compared to what would be required to complete IR > based witnesses because most of the heavy lifting is already done by > mutation tracking. > >> > >> Log replicas also makes it much more practical to realize the cost > savings of witnesses because log replicas have easier to characterize > resource consumption requirements (write rate * recovery/reconfiguration > time) and target a 10x improvement in write throughput. This makes knowing > how much capacity can be omitted safer and easier. > >> > >> Thanks, > >> Ariel > > > > > > -- Dmitry Konstantinov
Re: Welcome David Capwell as Cassandra PMC Member!
Congratulations David! On Mon, 28 Apr 2025 at 20:16, Abe Ratnofsky wrote: > Congrats David! > -- Dmitry Konstantinov
Re: CEP-15 Update
gt; isolated, I cannot imagine the cost to the community to removing this work >>>> would be very high. But, I do not intend to argue Accord’s case here. I >>>> will let you all decide. >>>> >>>> Please decide soon though, as it shapes our work planning. The positive >>>> reception so far had lead me to consider prioritising a move to trunk-first >>>> development within the next week or two, and the associated work that >>>> entails. However, if that was optimistic we will have to shift our plans. >>>> >>>> >>>> >>>> On 6 Mar 2025, at 20:16, Jordan West wrote: >>>> >>>> The work and effort in accord has been amazing. And I’m sure it sets a >>>> new standard for code quality and correctness testing which I’m also >>>> entirely behind. I also trust the folks working on it want to take it to >>>> the a fully production ready solution. But I’m worried about circumstances >>>> out of our control leaving us with a very complex feature that isn’t >>>> complete. >>>> >>>> I do have some questions. Could folks help me better understand why >>>> testing real workloads necessitates a merge (my understanding from the >>>> original reason is this is the impetus for why we would merge now)? Also I >>>> think the performance and scheme change caveats are rather large ones. One >>>> of accords promise was better performance and I think making schema changes >>>> with nodes down not being supported is a big gap. Could we have some >>>> criteria like “supports all the operations PaxosV2 supports” or “performs >>>> as well or better than PaxosV2 on [workload(s)]”? >>>> >>>> I understand waiting asks a lot of the authors in terms of baring the >>>> burden of a more complex merge. But I think we also need to consider what >>>> merging is asking the community to bear if the worst happens and we are >>>> unable to take the feature from its current state to something that can be >>>> widely used in production. >>>> >>>> >>>> Jordan >>>> >>>> >>>> On Wed, Mar 5, 2025 at 15:52 Blake Eggleston >>>> wrote: >>>> >>>>> +1 to merging it >>>>> >>>>> On Wed, Mar 5, 2025, at 12:22 PM, Patrick McFadin wrote: >>>>> >>>>> You have my +1 >>>>> >>>>> On Wed, Mar 5, 2025 at 12:16 PM Benedict wrote: >>>>> > >>>>> > Correct, these caveats should only apply to tables that have >>>>> opted-in to accord. >>>>> > >>>>> > On 5 Mar 2025, at 20:08, Jeremiah Jordan >>>>> wrote: >>>>> > >>>>> > >>>>> > So great to see all this hard work about to pay off! >>>>> > >>>>> > On the questions/concerns front, the only concern I would have >>>>> towards merging this to trunk is if any of the caveats apply when someone >>>>> is not using Accord. Assuming they only apply when the feature flag is >>>>> enabled, I see no reason not to get this merged into trunk once everyone >>>>> involved is happy with the state of it. >>>>> > >>>>> > -Jeremiah >>>>> > >>>>> > On Mar 5, 2025 at 12:15:23 PM, Benedict Elliott Smith < >>>>> bened...@apache.org> wrote: >>>>> >> >>>>> >> That depends on all of you lovely people :D >>>>> >> >>>>> >> I think we should have finished merging everything we want before >>>>> QA by ~Monday; certainly not much later. >>>>> >> >>>>> >> I think we have some upgrade and python dtest failures to address >>>>> as well. >>>>> >> >>>>> >> So it could be pretty soon if the community is supportive. >>>>> >> >>>>> >> On 5 Mar 2025, at 17:22, Patrick McFadin >>>>> wrote: >>>>> >> >>>>> >> >>>>> >> What is the timing for starting the merge process? I'm asking >>>>> because >>>>> >> >>>>> >> I have (yet another) presentation and this would be a cool update. >>>>> >> >>>>> >> >>>>> >> On Wed,
Re: Probable NP-completness of constraints satisfiability check
ere is ">" used twice and we just error out on this as >> it seems like a user can not make his mind which it is - > 10 or > 100? If >> an operator is used just once, that simplifies that a lot as well. >> >> We can also propose this problem as a subject of the Google Summer of >> Code project for students to deal with. I can imagine that it might be >> attractive for scholars who are trying to find a problem to solve with a >> lot of theory behind it while it is also practical. I do not know what >> plans Bernardo has with this as it is his brain-child. I just stumbled upon >> this topic and I am trying to drive it to some reasonable maturity but I do >> not plan to spend a significant amount of time on this afterwards ... >> >> On Tue, Feb 18, 2025 at 11:06 PM Dmitry Konstantinov >> wrote: >> >>> Hi, are you sure you want to go that deep? :-) >>> >>> https://en.wikipedia.org/wiki/Constraint_satisfaction_problem >>> https://en.wikipedia.org/wiki/2-satisfiability (an example what quite a >>> small change in constraints can change the picture dramatically: once we >>> moved from 3SAT to 2SAT - it is polynomial). >>> >>> https://www.cs.ox.ac.uk/activities/constraints/publications/ComplexityLanguages.pdf >>> >>> >>> Regards, >>> Dmitry >>> >>> On Tue, 18 Feb 2025 at 19:56, Štefan Miklošovič >>> wrote: >>> >>>> Sorry, CEP-42 >>>> >>>> On Tue, Feb 18, 2025 at 8:54 PM Štefan Miklošovič < >>>> smikloso...@apache.org> wrote: >>>> >>>>> Hi list, >>>>> >>>>> We are doing good progress together with Bernardo when it comes to >>>>> constraints which were merged recently (CEP-24). >>>>> >>>>> What I do now is that I try to "harden" it a little bit. If you think >>>>> about that, a user could do something like this (currently) >>>>> >>>>> ALTER TABLE ks.tb ALTER column1 CHECK column1 > 10 AND column1 < 5 >>>>> >>>>> If you take a look at this, this constraint is not satisfiable. There >>>>> is no such value for "column1" which would pass both checks. That means >>>>> that whatever a user would try to put there, such an insertion would fail. >>>>> >>>>> I have a POC which covers the most obvious cases so it fails when a >>>>> constraint is obviously not satisfiable, so we are not storing a check >>>>> which is impossible to pass. >>>>> >>>>> However, we are not completely sure how far we should go with this. >>>>> >>>>> The problem here is that, in theory, we could also support OR, for >>>>> example >>>>> >>>>> ALTER TABLE ks.tb ALTER column1 CHECK (column1 > 5 AND column < 10) OR >>>>> (column1 > 1000) >>>>> >>>>> You got it. >>>>> >>>>> The problem with this is that, when it is generalized, I can not check >>>>> if this is satisfiable, because this is basically a variation of SAT >>>>> problem, which is ... NP-complete. Right? In its most general form, is not >>>>> this a NP-complete problem? Or am I wrong? >>>>> >>>>> I do not think that we are going to solve NP-completeness of SAT here >>>>> :D so I just want to make sure that we are on the same page when it comes >>>>> to this and all we will ever support is just a simple AND (that is >>>>> complicated enough to verify if it is satisfiable already). >>>>> >>>>> What do you think about this? Is this all OK for you? >>>>> >>>>> Regards >>>>> >>>>> (1) https://en.wikipedia.org/wiki/Boolean_satisfiability_problem >>>>> >>>> >>> >>> -- >>> Dmitry Konstantinov >>> >> -- Dmitry Konstantinov
Re: Probable NP-completness of constraints satisfiability check
Hi, are you sure you want to go that deep? :-) https://en.wikipedia.org/wiki/Constraint_satisfaction_problem https://en.wikipedia.org/wiki/2-satisfiability (an example what quite a small change in constraints can change the picture dramatically: once we moved from 3SAT to 2SAT - it is polynomial). https://www.cs.ox.ac.uk/activities/constraints/publications/ComplexityLanguages.pdf Regards, Dmitry On Tue, 18 Feb 2025 at 19:56, Štefan Miklošovič wrote: > Sorry, CEP-42 > > On Tue, Feb 18, 2025 at 8:54 PM Štefan Miklošovič > wrote: > >> Hi list, >> >> We are doing good progress together with Bernardo when it comes to >> constraints which were merged recently (CEP-24). >> >> What I do now is that I try to "harden" it a little bit. If you think >> about that, a user could do something like this (currently) >> >> ALTER TABLE ks.tb ALTER column1 CHECK column1 > 10 AND column1 < 5 >> >> If you take a look at this, this constraint is not satisfiable. There is >> no such value for "column1" which would pass both checks. That means that >> whatever a user would try to put there, such an insertion would fail. >> >> I have a POC which covers the most obvious cases so it fails when a >> constraint is obviously not satisfiable, so we are not storing a check >> which is impossible to pass. >> >> However, we are not completely sure how far we should go with this. >> >> The problem here is that, in theory, we could also support OR, for example >> >> ALTER TABLE ks.tb ALTER column1 CHECK (column1 > 5 AND column < 10) OR >> (column1 > 1000) >> >> You got it. >> >> The problem with this is that, when it is generalized, I can not check if >> this is satisfiable, because this is basically a variation of SAT problem, >> which is ... NP-complete. Right? In its most general form, is not this a >> NP-complete problem? Or am I wrong? >> >> I do not think that we are going to solve NP-completeness of SAT here :D >> so I just want to make sure that we are on the same page when it comes to >> this and all we will ever support is just a simple AND (that is complicated >> enough to verify if it is satisfiable already). >> >> What do you think about this? Is this all OK for you? >> >> Regards >> >> (1) https://en.wikipedia.org/wiki/Boolean_satisfiability_problem >> > -- Dmitry Konstantinov
Re: Welcome Bernardo Botella as Cassandra Committer
Congrats Bernardo!! On Tue, 4 Mar 2025 at 10:58, Berenguer Blasi wrote: > Congrats! > > On 4/3/25 8:30, Štefan Miklošovič wrote: > > The Project Management Committee (PMC) for Apache Cassandra has > > invited Bernardo Botella to become a committer and we are pleased to > > announce that he has accepted. > > > > Please join us in welcoming Bernardo Botella to his new role and > > responsibility in our project community. > > > > Stefan Miklosovic > > > > On behalf of the Apache Cassandra PMC > -- Dmitry Konstantinov
Re: [DISCUSS] synchronisation of properties between Config.java and cassandra.yaml
>> https://docs.google.com/spreadsheets/d/11MOxhNqwE1tWP4ex2gzKG2pmeAWFaHDKo-CRp25h9BU/edit?gid=0#gid=0 We still have a lot of rows empty. I have added many default values and a Cassandra version when a parameter was introduced (to differentiate some recent parameters from old ones) based on source code but it would be nice to get a description for parameters from the authors as well as classification exposed/hidden. Maybe we should not wait for collecting info about all parameters and update what we have + use a threshold in the Ant validation task to fail when new missed parameters are added. The logic in the dev branch here https://issues.apache.org/jira/browse/CASSANDRA-20249 already supports a threshold. On Tue, 28 Jan 2025 at 00:04, Josh McKenzie wrote: > Good point re: the implications of parsing and durability in the face of > seeing unknown or missing parameters. I don't think widening the scope on > that would be ideal, especially considering the entire impetus for this > conversation is "we've misbehaved with our config and have a bunch of > undocumented stuff we're not sure is still useful, or what it's for". =/ > > On Mon, Jan 27, 2025, at 3:41 PM, Štefan Miklošovič wrote: > > "we take "unclaimed" items and move them to their own InternalConfig.java > or something" > > This is interesting. If we are meant to be still able to put these > properties into cassandra.yaml (even they are "internal ones") and they > would be just in InternalConfig.java for some basic separation of internal > / user-facing configuration, then we would need to have two yaml loaders: > > 1) the one as we have now which loads cassandra.yaml it into Config.java > 2) the second one which would load cassandra.yaml into InternalConfig.java > > For both cases, we could not fail when there are unrecognized properties > in cassandra.yaml while parsing it (1), because every loader, for > Config.java as well as InternalConfig.java, is parsing just some "subset" > of yaml. > > (1) > https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/config/YamlConfigurationLoader.java#L443-L444 > > If we just had "public InternalConfig internal = new InternalConfig" as a > field in Config.java, then this would lead to properties being effectively > renamed in cassandra.yaml like > > internal: > some_currently_internal_property: false > > instead of just > > some_currently_internal_property: false > > I do not think we want to have them renamed / under different > configuration sections in yaml. I get that they are "internal" etc but we > just don't know how / where it is used and deployed and just blindly > renaming them is not a good idea imho. > > On Mon, Jan 27, 2025 at 8:46 PM Josh McKenzie > wrote: > > > This may be an off-base comparison, but this reminds me of struggles we've > had getting to 0 failing unit tests before and the debates on fencing off a > snapshot of the current "failure set" so you can have a set point where no > further degradation is allowed in a primary data set. > > All of which is to say - maybe at the end of the spreadsheet, we take > "unclaimed" items and move them to their own InternalConfig.java or > something and add an ant target that a) disallows further addition to > InternalConfig.java w/out throwing an error / needing whitelist update, and > b) disallows further regression in the Config.java <-> cassandra.yaml > relationship for non-annotated fields. > > That way we can at least halt the progression of the disease even if we're > stymied on cleaning up some of the existing symptoms. > > On Mon, Jan 27, 2025, at 1:38 PM, Štefan Miklošovič wrote: > > Indeed, we need to balance that and thoughtfully choose what is going to > be added and what not. However, we should not hide something which is meant > to be tweaked by a user. The config is intimidating mostly because > everything is just in one file. I merely remember discussions a few years > ago which were about splitting cassandra.yaml into multiple files which > would be focused just on one subsystem / would cover some logically > isolated domain. > > Anyway, I think the main goal of this effort for now would be to at least > map where we are at. Some of them are genuinely missing. E.g. guardrails, > how is a user meant to know about that if it is not even documented ... > > On Mon, Jan 27, 2025 at 6:16 PM Chris lohfink wrote: > > Might be a bit of a balance between exposing what people actually are > likely to need to modify vs having a super intimidating config file. It's > already nearly 2000 lines. Personally I'd rather see some >
Re: Dropwizard/Codahale metrics deprecation in Cassandra server
Hi Jon
>> Is there a specific workload you're running where you're seeing it take
up a significant % of CPU time? Could you share some metrics, profile
data, or a workload so I can try to reproduce your findings?
Yes, I have shared the workload generation command (sorry, it is in
cassandra-stress, I have not yet adopted your tool but want to do it soon
:-) ), setup details and async profiler CPU profile in CASSANDRA-20250
<https://issues.apache.org/jira/browse/CASSANDRA-20250>
A summary:
- it is a plain insert-only workload to assert a max throughput capacity
for a single node: ./tools/bin/cassandra-stress "write n=10m" -rate
threads=100 -node myhost
- small amount of data per row is inserted, local SSD disks are used, so
CPU is a primary bottleneck in this scenario (while it is quite synthetic
in my real business cases CPU is a primary bottleneck as well)
- I used 5.1 trunk version (similar results I have for 5.0 version while
I was checking CASSANDRA-20165
<https://issues.apache.org/jira/browse/CASSANDRA-20165>)
- I enabled trie memetables + offheap objects mode
- I disabled compaction
- a recent nightly build is used for async-profiler
- my hardware is quite old: on-premise VM, Linux 4.18.0-240.el8.x86_64,
OpenJdk-11.0.26+4, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 16 cores
- link to CPU profile
<https://issues.apache.org/jira/secure/attachment/13074588/13074588_5.1_profile_cpu.html>
("codahale"
code: 8.65%)
- -XX:+DebugNonSafepoints option is enabled to improve the profile
precision
On Wed, 5 Mar 2025 at 12:38, Benedict Elliott Smith
wrote:
> Some quick thoughts of my own…
>
> === Performance ===
> - I have seen heap dumps with > 1GiB dedicated to metric counters. This
> patch should improve this, while opening up room to cut it further, steeply.
> - The performance improvement in relative terms for the metrics being
> replaced is rather dramatic - about 80%.. We can also improve this further.
> - Cheaper metrics (in terms of both cpu and memory) means we can readily
> have more of them, exposing finer-grained details. This is hard to
> understate the value of.
>
> === Reporting ===
> - We’re already non-standard for our most important metrics, because we
> had to replace the Codahale histogram years ago
> - We can continue implementing the Codahale interfaces, so that exporting
> libraries have minimal work to support us
> - We can probably push patches upstream to a couple of selected libraries
> we consider important
> - I would anyway also support picking a new reporting framework to
> support, but I would like us to do this with great care to avoid repeating
> our mistakes. I won’t have cycles to actually implement this, so it would
> be down to others to decide if they are willing to undertake this work
>
> I think the fallback option for now, however, is to abuse unsafe to allow
> us to override the implementation details of Codahale metrics. So we can
> decouple the performance discussion for now from the deprecation
> discussion, but I think we should have a target of deprecating
> Codahale/DropWizard for the reasons Dmitry outlines, however we decide to
> do it.
>
> On 4 Mar 2025, at 21:17, Jon Haddad wrote:
>
> I've got a few thoughts...
>
> On the performance side, I took a look at a few CPU profiles from past
> benchmarks and I'm seeing DropWizard taking ~ 3% of CPU time. Is there a
> specific workload you're running where you're seeing it take up a
> significant % of CPU time? Could you share some metrics, profile data, or
> a workload so I can try to reproduce your findings? In my testing I've
> found the majority of the overhead from metrics to come from JMX, not
> DropWizard.
>
> On the operator side, inventing our own metrics lib means risks making it
> harder to instrument Cassandra. There are libraries out there that allow
> you to tap into DropWizard metrics directly. For example, Sarma Pydipally
> did a presentation on this last year [1] based on some code I threw
> together.
>
> If you're planning on making it easier to instrument C* by supporting
> sending metrics to the OTel collector [2], then I could see the change
> being a net win as long as the perf is no worse than the status quo.
>
> It's hard to know the full extent of what you're planning and the impact,
> so I'll save any opinions till I know more about the plan.
>
> Thanks for bringing this up!
> Jon
>
> [1]
> https://planetcassandra.org/leaf/apache-cassandra-lunch-62-grafana-dashboard-for-apache-cassandra-business-platform-team/
> [2] https://opentelemetry.io/docs/collector/
>
> On Tue, Mar 4, 2025 at 12:40 PM Dmitry Konstantinov
> wrote:
>
>&g
Dropwizard/Codahale metrics deprecation in Cassandra server
Hi all, After a long conversation with Benedict and Maxim in CASSANDRA-20250 <https://issues.apache.org/jira/browse/CASSANDRA-20250> I would like to raise and discuss a proposal to deprecate Dropwizard/Codahale metrics usage in the next major release of Cassandra server and drop it in the following major release. Instead of it our own Java API and implementation should be introduced. For the next major release Dropwizard/Codahale API is still planned to support by extending Codahale implementations, to give potential users of this API enough time for transition. The proposal does not affect JMX API for metrics, it is only about local Java API changes within Cassandra server classpath, so it is about the cases when somebody outside of Cassandra server code relies on Codahale API in some kind of extensions or agents. Reasons: 1) Codahale metrics implementation is not very efficient from CPU and memory usage point of view. In the past we already replaced default Codahale implementations for Reservoir with our custom one and now in CASSANDRA-20250 <https://issues.apache.org/jira/browse/CASSANDRA-20250> we (Benedict and I) want to add a more efficient implementation for Counter and Meter logic. So, in total we do not have so much logic left from the original library (mostly a MetricRegistry as container for metrics) and the majority of logic is implemented by ourselves. We use metrics a lot along the read and write paths and they contribute a visible overhead (for example for plain write load it is about 9-11% according to async profiler CPU profile), so we want them to be highly optimized. >From memory perspective Counter and Meter are built based on LongAdder and they are quite heavy for the amounts which we create and use. 2) Codahale metrics does not provide any way to replace Counter and Meter implementations. There are no full functional interfaces for these entities + MetricRegistry has casts/checks to implementations and cannot work with anything else. I looked through the already reported issues and found the following similar and unsuccessful attempt to introduce interfaces for metrics: https://github.com/dropwizard/metrics/issues/2186 as well as other older attempts: https://github.com/dropwizard/metrics/issues/252 https://github.com/dropwizard/metrics/issues/264 https://github.com/dropwizard/metrics/issues/703 https://github.com/dropwizard/metrics/pull/487 https://github.com/dropwizard/metrics/issues/479 https://github.com/dropwizard/metrics/issues/253 So, the option to request an extensibility from Codahale metrics does not look real.. 3) It looks like the library is in maintenance mode now, 5.x version is on hold and many integrations are also not so alive. The main benefit to use Codahale metrics should be a huge amount of reporters/integrations but if we check carefully the list of reporters mentioned here: https://metrics.dropwizard.io/4.2.0/manual/third-party.html#reporters we can see that almost all of them are dead/archived. 4) In general, exposing other 3rd party libraries as our own public API frequently creates too many limitations and issues (Guava is another typical example which I saw previously, it is easy to start but later you struggle more and more). Does anyone have any questions or concerns regarding this suggestion? -- Dmitry Konstantinov
Re: Welcome Ekaterina Dimitrova as Cassandra PMC member
Congrats Ekaterina!! On Tue, 4 Mar 2025 at 23:25, Paulo Motta wrote: > Aloha, > > The Project Management Committee (PMC) for Apache Cassandra is delighted > to announce that Ekaterina Dimitrova has joined the PMC! > > Thanks a lot, Ekaterina, for everything you have done for the project all > these years. > > The PMC - Project Management Committee - manages and guides the direction > of the project, and is responsible for inviting new committers and PMC > members to steward the longevity of the project. > > See https://community.apache.org/pmc/responsibilities.html if you're > interested in learning more about the rights and responsibilities of PMC > members. > > Please join us in welcoming Ekaterina Dimitrova to her new role in our > project! > > Paulo, on behalf of the Apache Cassandra PMC > -- Dmitry Konstantinov
Re: CEP-15 Update
Thank you to all Accord and TCM contributors, it is really exciting to see a development of such huge and wonderful features moving forward and opening the door to the new Cassandra epoch! On Tue, 4 Mar 2025 at 20:45, Blake Eggleston wrote: > Thanks Benedict! > > I’m really excited to see accord reach this milestone, even with these > caveats. You seem to have left yourself off the list of contributors > though, even though you’ve been a central figure in its development :) So > thanks to all accord & tcm contributors, including Benedict, for making > this possible! > > On Tue, Mar 4, 2025, at 8:00 AM, Benedict Elliott Smith wrote: > > Hi everyone, > > It’s been exactly 3.5 years since the first commit to cassandra-accord. > Yes, really, it’s been that long. > > We will be starting to validate the feature against real workloads in the > near future, so we can’t sensibly push off merging much longer. The > following is a brief run-down of the state of play. There are no known > bugs, but there remain a number of caveats we will be incrementally > addressing in the run-up to a full release: > > [1] Accord is likely to be SLOW until further optimisations are implemented > [2] Schema changes have a number of hard edges > [3] Validation is ongoing, so there are likely still a number of bugs to > shake out > [4] Many operator visibility/tooling/documentation improvements are pending > > To expand a little: > > [1] As of the last experiment we conducted, accord’s throughput was poor - > also leading to higher LAN latencies. We have done no WAN experiments to > date, but the protocol guarantees should already achieve better round-trip > performance, in particular under contention. Improving throughput will be > the main focus of attention once we are satisfied the protocol is otherwise > stable, but our focus remains validation for the moment. > [2] Schema changes have not yet been well integrated with TCM. Dropping a > table for instance will currently cause problems if nodes are offline. > [3] We have a range of validations we are already performing against > cassandra-accord directly, and against its integration with Cassandra in > cep-15-accord. We have run hundreds of billions of simulated transactions, > and are still discovering some minor fault every few billion simulated > transactions or so. There remains a lot more simulated validation to > explore, as well as with real clusters serving real workloads. > [4] There are already a range of virtual tables for exploring internal > state in Accord, and reasonably good metric support. However, tracing is > not yet supported, and our metric and virtual table integrations need some > further development. > [5] There are also other edge cases to address such as ensuring we do not > reuse HLCs after restart, supporting ByteOrderPartitioner, and live > migration from/to Paxos is undergoing fine-tuning and validation; probably > there are some other things I am forgetting. > > Altogether the feature is fairly mature, despite these caveats. This is > the fruit of the labour of a long list of contributors, including Aleksey > Yeschenko, Alex Petrov, Ariel Weisberg, Blake Eggleston, Caleb Rackliffe > and David Capwell, and represents a huge undertaking. It also wouldn’t have > been possible without the work of Alex Petrov, Marcus Eriksson and Sam > Tunnicliffe on delivering transactional cluster metadata. I hope you will > join me in thanking them all for their contributions. > > Alex has also kindly produced some initial overview documentation for > developers, that can be found here: > https://github.com/apache/cassandra/blob/cep-15-accord/doc/modules/cassandra/pages/developing/accord/index.adoc. > This will be expanded as time permits. > > Does anyone have any questions or concerns? > > > -- Dmitry Konstantinov
Re: Dropwizard/Codahale metrics deprecation in Cassandra server
f your goal is to push write throughput as far as you can, something
>> I'm very much on board with, and is a key component to pushing density and
>> reducing cost. I'm spinning up a 5.0 cluster now to run a test, so I'll
>> run a load test similar to what you've done and try to reproduce your
>> results. I'll also review the JIRA to get more familiar with what you're
>> working on.
>> >
>> > Benedict: I agree with your line of thinking around optimizing the cost
>> of metrics. As we push both density and multi-tenancy, there's going to be
>> more and more demand for clusters with hundreds or thousands of tables.
>> Maybe tens of thousands. Reducing overhead for something that's O(N * M)
>> (multiple counters per table) will definitely be a welcome improvement.
>> There's always more stuff that's going to get in the way, but it's an
>> elephant and I appreciate every bite.
>> >
>> > My main concern with metrics isn't really compatibility, and I don't
>> have any real investment in DropWizard. I don't know if there's any real
>> value in putting in effort to maintain compatibility, but I'm just one
>> sample, so I won't make a strong statement here.
>> >
>> > It would be *very nice* we moved to metrics which implement the Open
>> Telemetry Metrics API [1], which I think solves multiple issues at once:
>> >
>> > * We can use either one of the existing implementations (OTel SDK) or
>> our own
>> > * We get a "free" upgrade that lets people tap into the OTel ecosystem
>> > * It paves the way for OTel traces with ZipKin [2] / Jaeger [3]
>> > * We can use the ubiquitous Otel instrumentation agent to send metrics
>> to the OTel collector, meaning people can collect at a much higher
>> frequency than today
>> > * OTel logging is a significant improvement over logback, you can
>> coorelate metrics + traces + logs together.
>> >
>> > Anyways, if the plan is to rip out something old and unmaintained and
>> replace with something new, I think there's a huge win to be had by
>> implementing the standard that everyone's using now.
>> >
>> > All this is very exciting and I appreciate the discussion!
>> >
>> > Jon
>> >
>> > [1] https://opentelemetry.io/docs/languages/java/api/
>> > [2] https://zipkin.io/
>> > [3] https://www.jaegertracing.io/
>> >
>> >
>> >
>> >
>> > On Wed, Mar 5, 2025 at 2:58 AM Dmitry Konstantinov
>> wrote:
>> >
>> > Hi Jon
>> >
>> > >> Is there a specific workload you're running where you're seeing it
>> take up a significant % of CPU time? Could you share some metrics, profile
>> data, or a workload so I can try to reproduce your findings?
>> > Yes, I have shared the workload generation command (sorry, it is in
>> cassandra-stress, I have not yet adopted your tool but want to do it soon
>> :-) ), setup details and async profiler CPU profile in CASSANDRA-20250
>> > A summary:
>> >
>> > it is a plain insert-only workload to assert a max throughput capacity
>> for a single node: ./tools/bin/cassandra-stress "write n=10m" -rate
>> threads=100 -node myhost
>> > small amount of data per row is inserted, local SSD disks are used, so
>> CPU is a primary bottleneck in this scenario (while it is quite synthetic
>> in my real business cases CPU is a primary bottleneck as well)
>> > I used 5.1 trunk version (similar results I have for 5.0 version while
>> I was checking CASSANDRA-20165)
>> > I enabled trie memetables + offheap objects mode
>> > I disabled compaction
>> > a recent nightly build is used for async-profiler
>> > my hardware is quite old: on-premise VM, Linux 4.18.0-240.el8.x86_64,
>> OpenJdk-11.0.26+4, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 16 cores
>> > link to CPU profile ("codahale" code: 8.65%)
>> > -XX:+DebugNonSafepoints option is enabled to improve the profile
>> precision
>> >
>> >
>> > On Wed, 5 Mar 2025 at 12:38, Benedict Elliott Smith <
>> bened...@apache.org> wrote:
>> >
>> > Some quick thoughts of my own…
>> >
>> > === Performance ===
>> > - I have seen heap dumps with > 1GiB dedicated to metric counters. This
>> patch should improve this, while opening up room to cut it further, steeply.
>> > - The performance improvement in relative terms for the metrics be
Re: Merging compaction improvements to 5.0
>>>>>>> wrote: >>>>>>> > >>>>>>> > Hey folks, >>>>>>> > >>>>>>> > Over the last 9 months Jordan and I have worked on CASSANDRA-15452 >>>>>>> [1]. The TL;DR is that we're internalizing a read ahead buffer to >>>>>>> allow us >>>>>>> to do fewer requests to disk during compaction and range reads. This >>>>>>> results in far fewer system calls (roughly 16x reduction) and on systems >>>>>>> with higher read latency, a significant improvement in compaction >>>>>>> throughput. We've tested several different EBS configurations and >>>>>>> found it >>>>>>> delivers up to a 10x improvement when read ahead is optimized to >>>>>>> minimize >>>>>>> read latency. I worked with AWS and the EBS team directly on this and >>>>>>> the >>>>>>> Best Practices for C* on EBS [2] I wrote for them. I've performance >>>>>>> tested >>>>>>> this patch extensively with hundreds of billions of operations across >>>>>>> several clusters and thousands of compactions. It has less of an >>>>>>> impact on >>>>>>> local NVMe, since the p99 latency is already 10-30x less than what you >>>>>>> see >>>>>>> on EBS (100micros vs 1-3ms), and you can do hundreds of thousands of >>>>>>> IOPS >>>>>>> vs a max of 16K. >>>>>>> > >>>>>>> > Related to this, Branimir wrote CASSANDRA-20092 [3], which >>>>>>> significantly improves compaction by avoiding reading the partition >>>>>>> index. >>>>>>> CASSANDRA-20092 has been merged to trunk already [4]. >>>>>>> > >>>>>>> > I think we should merge both of these patches into 5.0, as the >>>>>>> perf improvement should allow teams to increase density of EBS backed C* >>>>>>> clusters by 2-5x, driving cost way down. There's a lot of teams >>>>>>> running C* >>>>>>> on EBS now. I'm currently working with one that's bottlenecked on maxed >>>>>>> out EBS GP3 storage. I propose we merge both, because without >>>>>>> CASSANDRA-20092, we won't get the performance improvements in >>>>>>> CASSANDRA-15452 with BTI, only BIG format. I've tested BTI in other >>>>>>> situations and found it to be far more performant than BIG. >>>>>>> > >>>>>>> > If we were looking at a small win, I wouldn't care much, but since >>>>>>> these patches, combined with UCS, allows more teams to run C* on EBS at >>>>>>> > >>>>>>> 10TB / node, I think it's worth doing now. >>>>>>> > >>>>>>> > Thanks in advance, >>>>>>> > Jon >>>>>>> > >>>>>>> > [1] https://issues.apache.org/jira/browse/CASSANDRA-15452 >>>>>>> > [2] >>>>>>> https://aws.amazon.com/blogs/database/best-practices-for-running-apache-cassandra-with-amazon-ebs/ >>>>>>> > [3] https://issues.apache.org/jira/browse/CASSANDRA-20092 >>>>>>> > [4] >>>>>>> https://github.com/apache/cassandra/commit/3078aea1cfc70092a185bab8ac5dc8a35627330f >>>>>>> > >>>>>>> >>>>>> >> -- Dmitry Konstantinov
Re: New committers: Maxwell Guo and Dmitry Konstantinov
Thank you all. I am very happy to work with so many experienced and enthusiastic people! I will do my best to make Cassandra even better. I am very grateful to the people who spent their valuable time for reviews, advice and merging. Congrats Maxwell! :-) On Fri, 21 Feb 2025 at 20:07, Francisco Guerrero wrote: > Congrats to both of you! > > On 2025/02/20 17:47:05 Štefan Miklošovič wrote: > > The Project Management Committee (PMC) for Apache Cassandra > > has invited Maxwell Guo and Dmitry Konstantinov to become committers and > we > > are pleased > > to announce that they have accepted. > > > > Please join us in welcoming Maxwell Guo and Dmitry Konstantinov to their > > new role and > > responsibility in our project community. > > > > Stefan Miklosovic > > > > On behalf of the Apache Cassandra PMC > > > -- Dmitry Konstantinov
Re: Welcome Caleb Rackliffe to the PMC
Congratulations Caleb! On Fri, 21 Feb 2025 at 23:21, Yifan Cai wrote: > Congrats Caleb! > -- > *From:* Štefan Miklošovič > *Sent:* Friday, February 21, 2025 11:44:42 AM > *To:* dev@cassandra.apache.org > *Subject:* Re: Welcome Caleb Rackliffe to the PMC > > Congratulations Caleb, what a delivery (not only) in SAI department! > > On Thu, Feb 20, 2025 at 11:07 PM Jon Haddad > wrote: > > The PMC for Apache Cassandra is delighted to announce that Caleb Rackliffe > has joined it's membership! > > Caleb has been a member of the community for 10 years and is one of the > most active committers on the project. > > Please join us in welcoming Caleb to his new role! > > Jon > On behalf of the Cassandra PMC > > > -- Dmitry Konstantinov
Re: Welcome Abe Ratnofsky as Cassandra committer!
Congrats Abe! On Mon, 12 May 2025 at 17:46, Alex Petrov wrote: > Hello folks of the dev list, > > The Apache Cassandra PMC is very glad to announce that Abe Ratnofsky has > accepted our invitation to become a committer! > > Abe has been actively contributing to Cassandra itself, made outstanding > contributions to the Cassandra drivers, played a key role in the recently > accepted CEP-45 [1], and has been active in the community — including on > this mailing list and on Cassandra conferences and meetups. > > Please join us in congratulating and welcoming Abe! > > Alex Petrov > on behalf of the Apache Cassandra PMC > [1] > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-45:+Mutation+Tracking > > -- Dmitry Konstantinov
Re: [DISCUSS] Replace airlift/airline library with Picocli
> > goal); >> > - We can customize the help output so that the user doesn't notice >> > anything while using of the nodetool; >> > - The cli parser is the same as what we now do with cli arguments. >> > >> > This makes the library a good candidate, but leaves open the question >> > of changing the license of the lib in the future. However, these risks >> > are relatively small because the CLI library is not a monetizable >> > thing, as I believe. We can also mitigate the risks copying the lib to >> > sources, as it mentioned here: >> > https://picocli.info/#_getting_started >> > >> > >> > commons-cli >> > https://commons.apache.org/proper/commons-cli/ >> > >> > In terms of licenses, it is the easiest candidate for us to use as >> > it's under the asf, and in fact the library is already used in e.g. >> > BulkLoader, SSTableExpoert. >> > However, I'd like to point out the following disadvantages the library >> > has for our case: >> > - This is not a drop-in replacement for the airline commands, as the >> > lib does not have annotation for markup commands. We have to flesh out >> > all the options we have as java classes, or create our owns; >> > - Subcommands have to be supported manually, which requires extra >> > effort to adopt the cli parser (correct me if I'm wrong here). We have >> > at least several subcommands in the NodeTool e.g. cms describe, cms >> > snapshot; >> > - Apart from parsing the cli arguments, we need to manually initialize >> > the command class and set the input arguments we have. >> > >> > >> > JComannder >> > https://jcommander.org/ >> > >> > The library is licensed under the Apache 2.0 license, so the situation >> > is the same as for Picocli. Here I'd like to point out a few things I >> > encountered while prototyping: >> > - Unbinding the jmx-specific options from commands is quite tricky and >> > requires touching an internal API (which I won't do). Option >> > inheritance is not the way to go if we want to have a clear command >> > hierarchy regardless of the API used. >> > - We won't be able to inject a Logger (the Output class in terms of >> > NodeTool) or other abstractions (e.g. MBeans) directly into the >> > commands, because it doesn't support dependency injection. This is >> > also part of the activity of reusing the commands in other APIs, for >> > instance to execute them via CQL; >> > >> > More basic in comparison to the Picocli, focusing primarily on simple >> > annotation-based parsing and subcommands, and won't allow us to reuse >> > the commands outside of the cli. >> > >> > >> > airline2 >> > https://github.com/rvesse/airline >> > >> > The library is licensed under the Apache 2.0 license, and this is an >> > attempt to rebuild the original airline library. Currently, this is >> > not a drop-in replacement, as it has some minor API differences from >> > the original library. It is also not a better choice for us, as it has >> > the same drawbacks I mentioned for the previous alternatives, e.g. not >> > possible to unbind the specific options from the command and use them >> > only when commands are called from the cli. >> > >> > >> > >> > >> > [1] https://github.com/apache/cassandra/pull/2497/files >> > [2] https://lists.apache.org/thread/m9wfb3gdo9s210824c9rm2ojc9qv9412 >> > -- Dmitry Konstantinov
Re: [VOTE] CEP-46: Finish Transient Replication/Witnesses
+1 (nb) On Mon, 12 May 2025 at 17:21, Blake Eggleston wrote: > +1 > > On Mon, May 12, 2025, at 7:27 AM, Ariel Weisberg wrote: > > Hi dev@, > > I would like to start the voting for CEP-46: Finish Transient > Replication/Witnesses > > Proposal: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=353601959 > Discussion: > https://lists.apache.org/thread/0z0ms62s4sf235zbp4vv2ss2rpfs4cdd > > The vote will be open for 72 hours. A vote passes if there are at least 3 > binding +1s and no binding vetoes. > > Thanks, > Ariel > > -- Dmitry Konstantinov
Re: Welcome Bret McGuire as Cassandra PMC Member!
Congratulations Bret! On Mon, 19 May 2025 at 00:51, Bret McGuire wrote: > Thank you everyone for the kind words! > >- Bret - > > On Sat, May 17, 2025 at 11:44 PM Patrick McFadin > wrote: > >> Welcome Bret and congratulations! >> >> On Sat, May 17, 2025 at 10:46 AM Jaydeep Chovatia < >> chovatia.jayd...@gmail.com> wrote: >> >>> Congratulations! >>> >>> Jaydeep >>> >>> On May 16, 2025, at 6:47 PM, guo Maxwell wrote: >>> >>> >>> Congrats!! >>> >>> Jon Haddad 于2025年5月17日周六 08:10写道： >>> >>>> Congrats!! >>>> >>>> >>>> On Fri, May 16, 2025 at 4:13 PM Alexandre DUTRA >>>> wrote: >>>> >>>>> Great news, and so well deserved! Congratulations Bret! >>>>> >>>>> Le ven. 16 mai 2025 à 19:00, Jeremiah Jordan a >>>>> écrit : >>>>> >>>>>> Congrats Bret! >>>>>> >>>>>> On May 16, 2025 at 4:00:12 PM, Mick Semb Wever >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> The Project Management Committee (PMC) for Apache Cassandra is >>>>>>> delighted to announce that Bret McGuire has joined the PMC! Thank >>>>>>> you Bret for all your contributions to the project over the years, >>>>>>> especially with so many of the drivers. >>>>>>> >>>>>>> The PMC - Project Management Committee - manages and guides the >>>>>>> direction of the project, and is responsible for inviting new >>>>>>> committers and >>>>>>> PMC members to steward the longevity of the project. See >>>>>>> https://community.apache.org/pmc/responsibilities.html if you're >>>>>>> interested in learning more about the rights and responsibilities of >>>>>>> PMC members. >>>>>>> >>>>>>> Please join us in welcoming Bret McGuire to his new role in our >>>>>>> project! >>>>>>> >>>>>>> Mick, on behalf of the Apache Cassandra PMC >>>>>>> >>>>>> -- Dmitry Konstantinov
[DISCUSS][CASSANDRA-20681] Mark JDK 17 as production ready for Cassandra 5.0
Hi all, I've created a task to mark JDK 17 as production-ready for Cassandra 5.0 in our documentation - CASSANDRA-20681 <https://issues.apache.org/jira/browse/CASSANDRA-20681> Reasons: - Cassandra 5.0.x has had four bugfix releases and is stable (5.0.0 was released in September 2024, so it's been out for eight months). - I'm not aware of any open Cassandra issues specific to JDK 17. - Our CI has been running tests with JDK 17 on every commit for over a year. - JDK 17 is a mature LTS version. We already have a newer LTS (JDK 21), and JDK 24 has already been released. - There might be a vicious cycle here: we’re waiting for more user feedback, while users are waiting for the feature to be marked as non-experimental before adopting it more widely. Any objections to marking JDK 17 as production-ready for 5.0? Related threads where the topic about JDK17 status has been raised: - https://the-asf.slack.com/archives/CK23JSY2K/p1744313849439569 - https://the-asf.slack.com/archives/CJZLTM05A/p1746787244618429 - https://lists.apache.org/thread/np70b8ck21k0ojsjnotg3j9p2rrj29dp - https://stackoverflow.com/questions/79563058/java-17-support-for-cassandra-5 -- Dmitry Konstantinov
Re: [VOTE] Clarifying our JDK Support Policy
+1 (nb) There are the following pain points (mentioned in the discussion thread as well) related to this option but other options look even worse :-), so I think it is a fair cost: - potential necessity to update 3rd party dependencies in GA branches to support a newer JDK - deprecating of JDK features we use: the nearest one which I see is Unsafe methods removal (probably since JDK26) which will force us to switch to FFM API (which one is GA only since JDK 22, so it means to support older JDK versions we have to introduce some kind of conditional logic around it). On Wed, 18 Jun 2025 at 22:37, Jordan West wrote: > +1 > > On Wed, Jun 18, 2025 at 8:28 AM C. Scott Andreas > wrote: > >> +1 >> >> On Jun 18, 2025, at 11:08 AM, Aleksey Yeshchenko >> wrote: >> >> +1 >> >> On 18 Jun 2025, at 14:24, Josh McKenzie wrote: >> >> We're at 10 binding +1. Need 3 more to move this across the line. >> >> On Tue, Jun 17, 2025, at 6:58 PM, Joseph Lynch wrote: >> >> +1 >> >> On Tue, Jun 17, 2025 at 1:47 PM David Capwell wrote: >> >> +1 >> >> On Jun 17, 2025, at 3:44 AM, Štefan Miklošovič >> wrote: >> >> +1 >> >> On Fri, Jun 13, 2025 at 1:58 PM Josh McKenzie >> wrote: >> >> >> [DISCUSS] thread: >> https://lists.apache.org/thread/vr7j2ob92k6fbcwvlfo60l3scylzdbft >> >> Text to vote on: >> >> -- >> *[New LTS JDK Adoption]* >> >>- When a new JDK goes LTS, we prioritize: >> >> >>- Moving trunk to build, run, pass CI, and support language level of >> that JDK, dropping others >> - Adding support to *run* on that JDK to all supported GA >> releases, passing all CI using that JDK >>- These 2 operations must merge atomically. This allows us to >>preserve the contract of allowing like-to-like JDK's for a live C* upgrade >> >> *[Build, run, language level, Pre Commit CI, EOL]* >> >>- trunk builds, runs, has CI on, and supports the language level of 1 >>JDK at any given time (ideally latest LTS JDK) >> >> >>- Supported non-trunk GA branches: >> - build, run, pass CI, and support the language level of *the >> oldest JDK they are certified for* >> - Are supported to *run* on all LTS JDK's between their oldest >> supported and newest LTS supported by trunk >>- In the very rare case a feature would have to be removed due to JDK >>change (think UDF's scripting engine), we instead keep the maximum >>allowable JDK for that feature supported on trunk and subsequent releases. >>The feature is flagged for deprecate-then-remove or re-implementation >> based >>on dev ML discussion. If removed, we drop the required older JDK across >> all >>branches when the feature is removed. Supporting new LTS JDK's is >>considered higher priority than supporting features that JDK's are no >>longer compatible with, pending debate on the dev ML. >> >> >>- Dropping JDK support happens naturally as old releases go EOL. >> >> *[Post Commit JDK validation CI]* >> >>- Periodically we will run all CI pipelines for all *runtime* >>supported JDK's for that branch (cadence TBD) >>- We will add basic perf testing across all GA branches + their >>supported runtime JDK's with reference workloads from easy-cass-stress for >> a simple performance-based smoke test >> >> >> -- >> Vote structure: >> >>- Roll call >> >> <https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+Project+Governance> >>is 25. >>- Quorum: 13 (min required votes to qualify results) >>- Super-majority of participating votes in favor required to pass (9 >>in favor at 13 min votes, etc) >> >> Will leave the vote open for a week. >> >> >> >> -- Dmitry Konstantinov
