On the Sidecar discussion, while Sidecar is the preferred mechanism for the reasons described, the API is sufficiently generic enough to plugin a user implementations (essentially provide a list of sstables for a token range, and a mechanism to open an InputStream on any SSTable file component). A user could - for example - easily read from backup snapshots on a blob store.
> On Mar 26, 2023, at 1:04 PM, Josh McKenzie <jmcken...@apache.org> wrote: > > I want to second what Yifan's spoken to, specifically in terms of resource > isolation and availability. > > While the sidecar hasn't seen a ton of traffic and contributions since the > acceptance into the project and clearance of CEP-1, my intuition is that > that's due to the entrenched maturity of alternative sidecars out there since > we were slow as a project to build one, not out of a lack of demand for a > fully fleshed out sidecar. As functionality shows up in the ASF C* Sidecar, > there's going to be tension as operators are incentivized to run both their > bespoke sidecars they may be running alongside the ASF C* one. That's to be > expected and a necessary pain to take on during a transition that I > personally think is sorely needed. > > Having bulk operations for analytics and for reading and writing SSTables is > a pretty compelling carrot, and the more folks we can get running the sidecar > and the more contributors active on it, the more we can expect to see > interest and work show up there (repair coordination, REST API's, etc - all > of which we've talked about before on ML or slack). > > So I'm a strong +1 to it living in the sidecar. > > On Sat, Mar 25, 2023, at 11:05 AM, Brandon Williams wrote: >> Oh, that's significantly different and great news, please do! Thanks >> for the clarification, Doug! >> >> Kind Regards, >> Brandon >> >> On Fri, Mar 24, 2023 at 4:42 PM Doug Rohrer <droh...@apple.com >> <mailto:droh...@apple.com>> wrote: >> > >> > I agree that the analytics library will need to support vnodes. To be >> > clear, there’s nothing preventing the solution from working with vnodes >> > right now, and no assumptions about a 1:1 topology between a token and a >> > node. However, we don’t, today, have the ability to test vnode support >> > end-to-end. We are working towards that, however, and should be able to >> > remove the caveat from the released analytics library once we can properly >> > test vnode support. >> > If it helps, I can update the CEP to say something more like “Caveat: >> > Currently untested with vnodes - work is ongoing to remove this >> > limitation” if that helps? >> > >> > Doug >> > >> > > On Mar 24, 2023, at 11:43 AM, Brandon Williams <dri...@gmail.com >> > > <mailto:dri...@gmail.com>> wrote: >> > > >> > > On Fri, Mar 24, 2023 at 10:39 AM Jeremiah D Jordan >> > > <jeremiah.jor...@gmail.com <mailto:jeremiah.jor...@gmail.com>> wrote: >> > >> >> > >> I have concerns with the majority of this being in the sidecar and not >> > >> in the database itself. I think it would make sense for the server >> > >> side of this to be a new service exposed by the database, not in the >> > >> sidecar. That way it can be able to properly integrate with the >> > >> authentication and authorization apis, and to make it a first class >> > >> citizen in terms of having unit/integration tests in the main DB >> > >> ensuring no one breaks it. >> > > >> > > I don't think this can/should happen until it supports the database's >> > > default configuration with vnodes. >> >
