Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations
Great suggestion, Can external storage only be local storage media? Or can it be stored in any storage medium, such as object storage s3 ? We have previously implemented a tiered storage capability, that is, there are multiple storage media on one node, SSD, HDD, and data placement based on requests. After briefly browsing the proposals, it seems that there are some differences. Can you help to do some explain ？ Thanks 。 Claude Warren, Jr via dev 于2023年9月25日周一 14:49写道： > I have just filed CEP-36 [1] to allow for keyspace/table storage outside > of the standard storage space. > > There are two desires driving this change: > >1. The ability to temporarily move some keyspaces/tables to storage >outside the normal directory tree to other disk so that compaction can >occur in situations where there is not enough disk space for compaction and >the processing to the moved data can not be suspended. >2. The ability to store infrequently used data on slower cheaper >storage layers. > > I have a working POC implementation [2] though there are some issues still > to be solved and much logging to be reduced. > > I look forward to productive discussions, > Claude > > [1] > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory > > > -- you are the apple of my eye !
Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations
external storage can be any storage that you can produce a FileChannel for. There is an S3 library that does this so S3 is a definite possibility for storage in this solution. My example code only writes to a different directory on the same system. And there are a couple of places where I did not catch the file creation, those have to be found and redirected to the proxy location. I think that it may be necessary to have a java FileSystem object to make the whole thing work. The S3 library that I found also has an S3 FileSystem class. This solution uses the internal file name for for example an sstable name. The proxyfactory can examine the entire path and make a determination of where to read/write the file. So any determination that can be made based on the information in the file path can be implemented with this approach. There is no direct inspection of the data being written to determine routing. The only routing data are in the file name. I ran an inhouse demo where I showed that we could reroute a single table to a different storage while leaving the rest of the tables in the same keyspace alone. In discussing this with a colleague we hit upon the term "tiered nodes". If you can spread your data across the nodes so that some nodes get the infrequently used data (cold data) and other nodes receive the frequently used data (hot data) then the cold data nodes can use this process to store the data on S3 or similar systems. On Mon, Sep 25, 2023 at 10:45 AM guo Maxwell wrote: > Great suggestion, Can external storage only be local storage media? Or > can it be stored in any storage medium, such as object storage s3 ? > We have previously implemented a tiered storage capability, that is, there > are multiple storage media on one node, SSD, HDD, and data placement based > on requests. After briefly browsing the proposals, it seems that there are > some differences. Can you help to do some explain ？ Thanks 。 > > > Claude Warren, Jr via dev 于2023年9月25日周一 > 14:49写道： > >> I have just filed CEP-36 [1] to allow for keyspace/table storage outside >> of the standard storage space. >> >> There are two desires driving this change: >> >>1. The ability to temporarily move some keyspaces/tables to storage >>outside the normal directory tree to other disk so that compaction can >>occur in situations where there is not enough disk space for compaction >> and >>the processing to the moved data can not be suspended. >>2. The ability to store infrequently used data on slower cheaper >>storage layers. >> >> I have a working POC implementation [2] though there are some issues >> still to be solved and much logging to be reduced. >> >> I look forward to productive discussions, >> Claude >> >> [1] >> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations >> [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory >> >> >> > > -- > you are the apple of my eye ! >
Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations
- I think this is a great step forward. - Being able to move sstables around between tiers of storage is a feature Cassandra desperately needs, especially if one of those tiers is some sort of object storage - This looks like it's a foundational piece that enables that. Perhaps by a team that's already implemented this end to end? - Rather than building this piece by piece, I think it'd be awesome if someone drew up an end-to-end plan to implement tiered storage, so we can make sure we're discussing the whole final state, and not an implementation detail of one part of the final state? On Sun, Sep 24, 2023 at 11:49 PM Claude Warren, Jr via dev < dev@cassandra.apache.org> wrote: > I have just filed CEP-36 [1] to allow for keyspace/table storage outside > of the standard storage space. > > There are two desires driving this change: > >1. The ability to temporarily move some keyspaces/tables to storage >outside the normal directory tree to other disk so that compaction can >occur in situations where there is not enough disk space for compaction and >the processing to the moved data can not be suspended. >2. The ability to store infrequently used data on slower cheaper >storage layers. > > I have a working POC implementation [2] though there are some issues still > to be solved and much logging to be reduced. > > I look forward to productive discussions, > Claude > > [1] > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations > [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory > > >
Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations
"Rather than building this piece by piece, I think it'd be awesome if someone drew up an end-to-end plan to implement tiered storage, so we can make sure we're discussing the whole final state, and not an implementation detail of one part of the final state?" Do agree with jeff for this ~~~ If these feature can be supported in oss cassandra , I think it will be very popular, whether in a private deployment environment or a public cloud service (our experience can prove it). In addition, it is also a cost-cutting option for users too Jeff Jirsa 于2023年9月26日周二 00:11写道： > > - I think this is a great step forward. > - Being able to move sstables around between tiers of storage is a feature > Cassandra desperately needs, especially if one of those tiers is some sort > of object storage > - This looks like it's a foundational piece that enables that. Perhaps by > a team that's already implemented this end to end? > - Rather than building this piece by piece, I think it'd be awesome if > someone drew up an end-to-end plan to implement tiered storage, so we can > make sure we're discussing the whole final state, and not an implementation > detail of one part of the final state? > > > > > > > On Sun, Sep 24, 2023 at 11:49 PM Claude Warren, Jr via dev < > dev@cassandra.apache.org> wrote: > >> I have just filed CEP-36 [1] to allow for keyspace/table storage outside >> of the standard storage space. >> >> There are two desires driving this change: >> >>1. The ability to temporarily move some keyspaces/tables to storage >>outside the normal directory tree to other disk so that compaction can >>occur in situations where there is not enough disk space for compaction >> and >>the processing to the moved data can not be suspended. >>2. The ability to store infrequently used data on slower cheaper >>storage layers. >> >> I have a working POC implementation [2] though there are some issues >> still to be solved and much logging to be reduced. >> >> I look forward to productive discussions, >> Claude >> >> [1] >> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations >> [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory >> >> >> -- you are the apple of my eye !
Re: [DISCUSS] CEP-36: A Configurable ChannelProxy to alias external storage locations
My intention is to develop an S3 storage system using https://github.com/carlspring/s3fs-nio There are several issues yet to be solved: 1. There are some internal calls that create files in the table directory that do not use the channel proxy. I believe that these are making calls on File objects. I think those File objects are Cassandra File objects not Java I/O File objects, but am unsure. 2. Determine if the carlspring s3fs-nio library will be performant enough to work in the long run. There may be issues with it: 1. Downloading entire files before using them rather than using views into larger remotely stored files. 2. Requiring a complete file to upload rather than using the partial upload capability of the S3 interface. On Tue, Sep 26, 2023 at 4:11 AM guo Maxwell wrote: > "Rather than building this piece by piece, I think it'd be awesome if > someone drew up an end-to-end plan to implement tiered storage, so we can > make sure we're discussing the whole final state, and not an implementation > detail of one part of the final state?" > > Do agree with jeff for this ~~~ If these feature can be supported in oss > cassandra , I think it will be very popular, whether in a private > deployment environment or a public cloud service (our experience can prove > it). In addition, it is also a cost-cutting option for users too > > Jeff Jirsa 于2023年9月26日周二 00:11写道： > >> >> - I think this is a great step forward. >> - Being able to move sstables around between tiers of storage is a >> feature Cassandra desperately needs, especially if one of those tiers is >> some sort of object storage >> - This looks like it's a foundational piece that enables that. Perhaps by >> a team that's already implemented this end to end? >> - Rather than building this piece by piece, I think it'd be awesome if >> someone drew up an end-to-end plan to implement tiered storage, so we can >> make sure we're discussing the whole final state, and not an implementation >> detail of one part of the final state? >> >> >> >> >> >> >> On Sun, Sep 24, 2023 at 11:49 PM Claude Warren, Jr via dev < >> dev@cassandra.apache.org> wrote: >> >>> I have just filed CEP-36 [1] to allow for keyspace/table storage outside >>> of the standard storage space. >>> >>> There are two desires driving this change: >>> >>>1. The ability to temporarily move some keyspaces/tables to storage >>>outside the normal directory tree to other disk so that compaction can >>>occur in situations where there is not enough disk space for compaction >>> and >>>the processing to the moved data can not be suspended. >>>2. The ability to store infrequently used data on slower cheaper >>>storage layers. >>> >>> I have a working POC implementation [2] though there are some issues >>> still to be solved and much logging to be reduced. >>> >>> I look forward to productive discussions, >>> Claude >>> >>> [1] >>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-36%3A+A+Configurable+ChannelProxy+to+alias+external+storage+locations >>> [2] https://github.com/Claudenw/cassandra/tree/channel_proxy_factory >>> >>> >>> > > -- > you are the apple of my eye ! >
