Hi Stefan, That idea is not related to this CEP which is about the file formats of the sstables, not file system access. But you should take a look at the work recently committed in https://issues.apache.org/jira/browse/CASSANDRA-16926 to switch to using java.nio.file.Path for file access. This should allow the use of a file system provider to access files which could be the basis for work to load the files from S3.
-Jeremiah On Fri, Oct 22, 2021 at 4:07 AM Stefan Miklosovic < stefan.mikloso...@instaclustr.com> wrote: > One point I would like to add to this; I was already looking into how > to extend this but what I saw in SSTableReader was that it is very > much "file system oriented". There was not any possibility to actually > hook something like that there. I think what importing does is that it > will use SSTableReader / Writer stuff so I think that the modification > of these classes to accommodate this idea would be necessary. > > On Fri, 22 Oct 2021 at 11:02, Stefan Miklosovic > <stefan.mikloso...@instaclustr.com> wrote: > > > > Hi Jacek, > > > > Thanks for taking the lead on this. > > > > There was importing of SSTables introduced in 4.0 via > > StorageService#importNewSSTables. The "problem" with this is that > > SSTables need to be physically located at disk so Cassandra can read > > them. If a backup is taken and SSTables are uploaded to, for example, > > S3 bucket, then upon restore, all these SSTables need to be downloaded > > first and then imported. What about downloading them / importing them > > directly from S3? Or any custom source for that matter? Importing of > > SSTables is a very nice feature in 4.0, we do not need to copy / hard > > link / refresh, it is all handled internally. > > > > I am not sure if your work is related to this idea but I would > > appreciate it if this is pluggable as well for the sake of simplicity > > and effectiveness as we would not have to download all sstables before > > importing them. > > > > If it is not related, feel free to skip that completely and I guess I > > would have to try to push that forward myself. > > > > Regards > > > > > > On Fri, 22 Oct 2021 at 10:24, Jacek Lewandowski > > <lewandowski.ja...@gmail.com> wrote: > > > > > > I'd like to start a discussion about SSTable format API proposal > (CEP-17) > > > > > > Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056 > > > CEP: > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API > > > > > > Thanks, > > > Jacek > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
