Hi Stefan,
That idea is not related to this CEP which is about the file formats of the
sstables, not file system access.  But you should take a look at the work
recently committed in https://issues.apache.org/jira/browse/CASSANDRA-16926
to switch to using java.nio.file.Path for file access.  This should allow
the use of a file system provider to access files which could be the basis
for work to load the files from S3.

-Jeremiah

On Fri, Oct 22, 2021 at 4:07 AM Stefan Miklosovic <
stefan.mikloso...@instaclustr.com> wrote:

> One point I would like to add to this; I was already looking into how
> to extend this but what I saw in SSTableReader was that it is very
> much "file system oriented". There was not any possibility to actually
> hook something like that there. I think what importing does is that it
> will use SSTableReader / Writer stuff so I think that the modification
> of these classes to accommodate this idea would be necessary.
>
> On Fri, 22 Oct 2021 at 11:02, Stefan Miklosovic
> <stefan.mikloso...@instaclustr.com> wrote:
> >
> > Hi Jacek,
> >
> > Thanks for taking the lead on this.
> >
> > There was importing of SSTables introduced in 4.0 via
> > StorageService#importNewSSTables. The "problem" with this is that
> > SSTables need to be physically located at disk so Cassandra can read
> > them. If a backup is taken and SSTables are uploaded to, for example,
> > S3 bucket, then upon restore, all these SSTables need to be downloaded
> > first and then imported. What about downloading them / importing them
> > directly from S3? Or any custom source for that matter? Importing of
> > SSTables is a very nice feature in 4.0, we do not need to copy / hard
> > link / refresh, it is all handled internally.
> >
> > I am not sure if your work is related to this idea but I would
> > appreciate it if this is pluggable as well for the sake of simplicity
> > and effectiveness as we would not have to download all sstables before
> > importing them.
> >
> > If it is not related, feel free to skip that completely and I guess I
> > would have to try to push that forward myself.
> >
> > Regards
> >
> >
> > On Fri, 22 Oct 2021 at 10:24, Jacek Lewandowski
> > <lewandowski.ja...@gmail.com> wrote:
> > >
> > > I'd like to start a discussion about SSTable format API proposal
> (CEP-17)
> > >
> > > Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056
> > > CEP:
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API
> > >
> > > Thanks,
> > > Jacek
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Reply via email to