I think it would be a mistake to combine the Memtable with CommitLog; several systems use CommitLog-like functionality, and in the medium term I think these would benefit from a unified system, that Memtables may opt to register with. It might make sense to give the Memtable the choice over whether a Memtable write is persisted to this shared facility, but that’s different from merging the two conceptually.
I may look into producing a CEP on this evolution sometime in the next few months, but just a heads up about my thoughts on the topic, and to reach out if you plan your own evolution of this stuff. From: Joshua McKenzie <jmcken...@apache.org> Date: Tuesday, 20 July 2021 at 18:36 To: dev@cassandra.apache.org <dev@cassandra.apache.org> Subject: Re: [DISCUSS] CEP-11: Pluggable memtable implementations +1 to the idea. In general, I think we need to make up our mind as to whether we consider the Memtable and CommitLog one logical entity (As stated in the CEP: "Conceptually these two pieces of the storage engine form one component — the LSM buffer of Cassandra, and as such it makes a lot of sense to bundle them together. "), or whether we want to further untangle those two components from an architectural perspective which we started down that road on with the pluggable storage engine work. The interface as drafted codifies the idea that a Memtable should have an opinion about how a CommitLog does its business (default boolean writesShouldSkipCommitLog()) which makes sense if our design goal is to keep those two things interdependent. I advocate for further separating them but suspect that's a debate better had on JIRA or slack than the CEP thread, just figured I'd bring it up since it's not yet clear to me whether that's a pre or post CEP discussion (specific details of interfaces, etc). Lots of quality work obviously went into this from a bunch of folks - thanks Branimir! ~Josh On Tue, Jul 20, 2021 at 6:20 AM bened...@apache.org <bened...@apache.org> wrote: > +1. I haven’t looked in detail at the API that’s been proposed, but I’m > very much in favour of the work to support this, and the introduction of > the newly proposed implementations. > > In particular, really happy to see somebody finally finish up C-7282! I > look forward to seeing how the different approaches compare. > > > From: Branimir Lambov <branimir.lam...@datastax.com> > Date: Tuesday, 20 July 2021 at 11:11 > To: dev@cassandra.apache.org <dev@cassandra.apache.org> > Subject: [DISCUSS] CEP-11: Pluggable memtable implementations > Proposal for a mechanism for plugging in memtable implementations: > > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-11%3A+Pluggable+memtable+implementations > > The proposal supports using custom memtable implementations to support > development and testing of improved alternatives, but also enables a > broader definition of "memtable" to better support more advanced use cases > like persistent memory. To this end, memtable implementations are given > control over flushing and storing data in the commit log, enabling > solutions that implement their own durability mechanisms and live much > longer than their classical counterparts. Taken to the extreme, this also > enables memtables that never flush (in other words, alternative storage > engines) in a minimally-invasive manner. > > I am curious to hear your thoughts on the proposal. > > Regards, > Branimir >
