Paging in bytes (CASSANDRA-11745)
Hi I wanted to start working on extending the paging mechanism to support memory based limits. My question is whether such a change requires a CEP or discussion in Jira is enough? Thanks, - - -- --- - - Jacek Lewandowski
Re: Paging in bytes (CASSANDRA-11745)
I personally think this change should be small enough to just be done via jira without introducing a CEP. But since paging generally has seen some problems, it'd be good to understand the design before flushing it out in code. One of the ways where we probably should be considerate is the paging state. Paging state is passed from the coordinator to client, and then from the client to a (potentially different) coordinator. If the coordinator speaks the internode protocol of a different version, this can potentially lead to problems. That said, if we have transactional metadata implemented and can reliably track versions in the cluster, and switch between epochs in which an effective minimal version for all coordinators supports a specific version of client state, this should not be a problem at all. On Fri, Oct 22, 2021 at 9:07 AM Jacek Lewandowski < lewandowski.ja...@gmail.com> wrote: > Hi > > I wanted to start working on extending the paging mechanism to support > memory based limits. My question is whether such a change requires a CEP or > discussion in Jira is enough? > > Thanks, > - - -- --- - - > Jacek Lewandowski > -- alex p
Permission to create a CEP for pluggable SSTable formats
I'd like to add a CEP for pluggable SSTable format but I don't have permission to create one - could anybody help me with that? Jacek - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Permission to create a CEP for pluggable SSTable formats
Noting here that I responded on the #cassandra-dev channel so someone could grant Jacek edit access to the cwiki. Cheers!
[DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
I'd like to start a discussion about SSTable format API proposal (CEP-17) Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056 CEP: https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API Thanks, Jacek - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
Hi Jacek, Thanks for taking the lead on this. There was importing of SSTables introduced in 4.0 via StorageService#importNewSSTables. The "problem" with this is that SSTables need to be physically located at disk so Cassandra can read them. If a backup is taken and SSTables are uploaded to, for example, S3 bucket, then upon restore, all these SSTables need to be downloaded first and then imported. What about downloading them / importing them directly from S3? Or any custom source for that matter? Importing of SSTables is a very nice feature in 4.0, we do not need to copy / hard link / refresh, it is all handled internally. I am not sure if your work is related to this idea but I would appreciate it if this is pluggable as well for the sake of simplicity and effectiveness as we would not have to download all sstables before importing them. If it is not related, feel free to skip that completely and I guess I would have to try to push that forward myself. Regards On Fri, 22 Oct 2021 at 10:24, Jacek Lewandowski wrote: > > I'd like to start a discussion about SSTable format API proposal (CEP-17) > > Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056 > CEP: > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API > > Thanks, > Jacek > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
One point I would like to add to this; I was already looking into how to extend this but what I saw in SSTableReader was that it is very much "file system oriented". There was not any possibility to actually hook something like that there. I think what importing does is that it will use SSTableReader / Writer stuff so I think that the modification of these classes to accommodate this idea would be necessary. On Fri, 22 Oct 2021 at 11:02, Stefan Miklosovic wrote: > > Hi Jacek, > > Thanks for taking the lead on this. > > There was importing of SSTables introduced in 4.0 via > StorageService#importNewSSTables. The "problem" with this is that > SSTables need to be physically located at disk so Cassandra can read > them. If a backup is taken and SSTables are uploaded to, for example, > S3 bucket, then upon restore, all these SSTables need to be downloaded > first and then imported. What about downloading them / importing them > directly from S3? Or any custom source for that matter? Importing of > SSTables is a very nice feature in 4.0, we do not need to copy / hard > link / refresh, it is all handled internally. > > I am not sure if your work is related to this idea but I would > appreciate it if this is pluggable as well for the sake of simplicity > and effectiveness as we would not have to download all sstables before > importing them. > > If it is not related, feel free to skip that completely and I guess I > would have to try to push that forward myself. > > Regards > > > On Fri, 22 Oct 2021 at 10:24, Jacek Lewandowski > wrote: > > > > I'd like to start a discussion about SSTable format API proposal (CEP-17) > > > > Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056 > > CEP: > > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API > > > > Thanks, > > Jacek > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
Hi Stefan, That idea is not related to this CEP which is about the file formats of the sstables, not file system access. But you should take a look at the work recently committed in https://issues.apache.org/jira/browse/CASSANDRA-16926 to switch to using java.nio.file.Path for file access. This should allow the use of a file system provider to access files which could be the basis for work to load the files from S3. -Jeremiah On Fri, Oct 22, 2021 at 4:07 AM Stefan Miklosovic < stefan.mikloso...@instaclustr.com> wrote: > One point I would like to add to this; I was already looking into how > to extend this but what I saw in SSTableReader was that it is very > much "file system oriented". There was not any possibility to actually > hook something like that there. I think what importing does is that it > will use SSTableReader / Writer stuff so I think that the modification > of these classes to accommodate this idea would be necessary. > > On Fri, 22 Oct 2021 at 11:02, Stefan Miklosovic > wrote: > > > > Hi Jacek, > > > > Thanks for taking the lead on this. > > > > There was importing of SSTables introduced in 4.0 via > > StorageService#importNewSSTables. The "problem" with this is that > > SSTables need to be physically located at disk so Cassandra can read > > them. If a backup is taken and SSTables are uploaded to, for example, > > S3 bucket, then upon restore, all these SSTables need to be downloaded > > first and then imported. What about downloading them / importing them > > directly from S3? Or any custom source for that matter? Importing of > > SSTables is a very nice feature in 4.0, we do not need to copy / hard > > link / refresh, it is all handled internally. > > > > I am not sure if your work is related to this idea but I would > > appreciate it if this is pluggable as well for the sake of simplicity > > and effectiveness as we would not have to download all sstables before > > importing them. > > > > If it is not related, feel free to skip that completely and I guess I > > would have to try to push that forward myself. > > > > Regards > > > > > > On Fri, 22 Oct 2021 at 10:24, Jacek Lewandowski > > wrote: > > > > > > I'd like to start a discussion about SSTable format API proposal > (CEP-17) > > > > > > Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056 > > > CEP: > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API > > > > > > Thanks, > > > Jacek > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: Permission to create a CEP for pluggable SSTable formats
And this looks like it was done for anyone that finds this later. On Fri, Oct 22, 2021 at 12:41 AM Erick Ramirez wrote: > Noting here that I responded on the #cassandra-dev channel so someone could > grant Jacek edit access to the cwiki. Cheers! >
[DISCUSS] CEP-18: Improving Modularity
Hi All, As has been seen with the work already started in CEP-10, increasing the modularity of our subsystems can improve their testability, and also the ability to try new implementations without breaking things. Our team has been working on doing this and CEP-18 has been created to propose adding more modularity to a few different subsystems. https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-18%3A+Improving+Modularity CASSANDRA-17044 has already been created for Schema Storage changes related to this work and more JIRAs and PRs are to follow for the other subsystems proposed in the CEP. Thanks, -Jeremiah Jordan
