Jaydeep, do you have any metrics on your clusters comparing them before and after introducing repair scheduling into the Cassandra process?
On 2024/10/21 16:57:57 "J. D. Jordan" wrote: > Sounds good. Just wanted to bring it up. I agree that the scheduling bit is > pretty light weight and the ideal would be to bring the whole of the repair > external, which is a much bigger can of worms to open. > > > > -Jeremiah > > > > > On Oct 21, 2024, at 11:21 AM, Chris Lohfink <clohfin...@gmail.com> wrote: > > > > > > > > > > > > I actually think we should be looking at how we can move things out of the > > database process. > > > > > > > > > > > > While worth pursuing, I think we would need a different CEP just to figure > > out how to do that. Not only is there a lot of infrastructure difficulty in > > running multi process, the inter app communication needs to be figured out > > better then JMX. Even the sidecar we dont have a solid story on how to > > ensure both are running or anything yet. It's up to each app owner to figure > > it out. Once we have a good thing in place I think we can start moving > > compactions, repairs, etc out of the database. Even then it's the _repairs_ > > that is expensive, not the scheduling. > > > > > > > > > > On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan > > <[jeremiah.jor...@gmail.com](mailto:jeremiah.jor...@gmail.com)> wrote: > > > > > > >> I love the idea of a repair service being there by default for an install > of C*. My main concern here is that it is putting more services into the main > database process. I actually think we should be looking at how we can move > things out of the database process. The C* process being a giant monolith has > always been a pain point. Is there anyway it makes sense for this to be an > external process rather than a new thread pool inside the C* process? > > >> > > >> > > > >> > > >> -Jeremiah Jordan > > >> > > >> > > > >> > > >> On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever > <[m...@apache.org](mailto:m...@apache.org)> wrote: > > > >> > > >>> > > > >>> > > >>> This is looking strong, thanks Jaydeep. > > >>> > > >>> > > > >>> > > >>> I would suggest folk take a look at the design doc and the PR in the CEP. > A lot is there (that I have completely missed). > > >>> > > >>> > > > >>> > > >>> I would especially ask all authors of prior art (Reaper, DSE nodesync, > ecchronos) to take a final review of the proposal > > > >>> > > >>> > > > >>> > > >>> Jaydeep, can we ask for a two week window while we reach out to these > people ? There's a lot of prior art in this space, and it feels like we're in > a good place now where it's clear this has legs and we can use that to bring > folk in and make sure there's no remaining blindspots. > > >>> > > >>> > > > >>> > > >>> > > > >>> > > >>> On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia > <[chovatia.jayd...@gmail.com](mailto:chovatia.jayd...@gmail.com)> wrote: > > > >>> > > >>>> Sorry, there is a typo in the CEP-37 link; here is the correct > [link](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution) > > >>>> > > >>>> > > > >>>> > > >>>> > > > >>>> > > >>>> On Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia > <[chovatia.jayd...@gmail.com](mailto:chovatia.jayd...@gmail.com)> wrote: > > > >>>> > > >>>>> First, thank you for your patience while we strengthened the CEP-37. > > >>>>> > > >>>>> > > > >>>>> > > >>>>> Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie, > Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online > discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37) to > come up with the best possible design that not only significantly simplifies > repair operations but also includes the most common features that everyone > will benefit from running at Scale. > > >>>>> > > >>>>> For example, > > >>>>> > > >>>>> * Apache Cassandra must be capable of running multiple repair types, > such as Full, Incremental, Paxos, and Preview - so the framework should be > easily extendable with no additional overhead from the operator’s point of > view. > > >>>>> > > >>>>> * An easy way to extend the token-split calculation algorithm with a > default implementation should exist. > > >>>>> > > >>>>> * Running incremental repair reliably at Scale is pretty challenging, > so we need to place safeguards, such as migration/rollback w/o restart and > stopping incremental repair automatically if the disk is about to get full. > > >>>>> > > >>>>> > > >>>>> > > >>>>> We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is > now officially ready for review after multiple rounds of design, testing, code > reviews, documentation reviews, and, more importantly, validation that it runs > at Scale! > > >>>>> > > >>>>> > > > >>>>> > > >>>>> Some facts about CEP-37. > > >>>>> > > >>>>> * Multiple members have verified all aspects of CEP-37 numerous times. > > >>>>> > > >>>>> * The design proposed in CEP-37 has been thoroughly tried and tested > on an immense scale (hundreds of unique Cassandra clusters, tens of thousands > of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source > for more than five years; please see more details[ > here](https://www.uber.com/en-US/blog/how-uber-optimized-cassandra-operations- > at-scale/). > > >>>>> > > >>>>> * The following > [presentation](https://docs.google.com/presentation/d/1Zilww9c7LihHULk_ckErI2s4XbObxjWknKqRtbvHyZc/edit#slide=id.g30a4fd4fcf7_0_13) > highlights the rigorous applied to CEP-37, which was given during last week’s > Apache Cassandra Bay Area [Meetup](https://www.meetup.com/apache-cassandra- > bay-area/events/303469006/), > > >>>>> > > >>>>> > > > > > >>>>> > > >>>>> Since things are massively overhauled, we believe it is almost ready for > a final pass pre-VOTE. We would like you to please review the > [CEP-37](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+Apache+Cassandra+Unified+Repair+Solution\)) > and the associated detailed design > [doc](https://docs.google.com/document/d/1CJWxjEi- > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0). > > >>>>> > > >>>>> > > > >>>>> > > >>>>> Thank you everyone! > > >>>>> > > >>>>> Chris, Andy, Josh, Dinesh, Kristijonas, and Jaydeep > > >>>>> > > >>>>> > > > > > >>>>> > > >>>>> > > > >>>>> > > >>>>> On Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie > <[jmcken...@apache.org](mailto:jmcken...@apache.org)> wrote: > > > >>>>> > > >>>>>> __ > > >>>>>> > > >>>>>> Not quite; finishing touches on the CEP and design doc are in flight > (as of last / this week). > > > >>>>>> > > >>>>>> > > > >>>>>> > > >>>>>> Soon(tm). > > >>>>>> > > >>>>>> > > > >>>>>> > > >>>>>> On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote: > > > >>>>>> > > >>>>>>> Is this CEP ready for a VOTE thread? > <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution> > > > > >>>>>>> > > >>>>>>> > > > >>>>>>> > > >>>>>>> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia > <[chovatia.jayd...@gmail.com](mailto:chovatia.jayd...@gmail.com)> wrote: > > > >>>>>>> > > >>>>>>>> Thanks, Josh. I've just updated the > [CEP](https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution) > and included all the solutions you mentioned below. > > > >>>>>>>> > > >>>>>>>> > > > >>>>>>>> > > >>>>>>>> Jaydeep > > > >>>>>>>> > > >>>>>>>> > > > >>>>>>>> > > >>>>>>>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie > <[jmcken...@apache.org](mailto:jmcken...@apache.org)> wrote: > > > >>>>>>>> > > >>>>>>>>> __ > > > >>>>>>>>> > > >>>>>>>>> Very late response from me here (basically necro'ing this thread). > > > >>>>>>>>> > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> I think it'd be useful to get this condensed into a CEP that we can > then discuss in that format. It's clearly something we all agree we need and > having an implementation that works, even if it's not in your preferred > execution domain, is vastly better than nothing IMO. > > > >>>>>>>>> > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> I don't have cycles (nor background ;) ) to do that, but it sounds > like you do Jaydeep given the implementation you have on a private fork + > design. > > > >>>>>>>>> > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> A non-exhaustive list of things that might be useful incorporating > into or referencing from a CEP: > > > >>>>>>>>> > > >>>>>>>>> Slack thread: <https://the- > asf.slack.com/archives/CK23JSY2K/p1690225062383619> > > > >>>>>>>>> > > >>>>>>>>> Joey's old C* ticket: > <https://issues.apache.org/jira/browse/CASSANDRA-14346> > > > >>>>>>>>> > > >>>>>>>>> Even older automatic repair scheduling: > <https://issues.apache.org/jira/browse/CASSANDRA-10070> > > > >>>>>>>>> > > >>>>>>>>> Your design gdoc: <https://docs.google.com/document/d/1CJWxjEi- > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0> > > > >>>>>>>>> > > >>>>>>>>> PR with automated repair: > <https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c> > > > > >>>>>>>>> > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> My intuition is that we're all basically in agreement that this is > something the DB needs, we're all willing to bikeshed for our personal > preference on where it lives and how it's implemented, and at the end of the > day, code talks. I don't think anyone's said they'll die on the hill of > implementation details, so that feels like CEP time to me. > > > >>>>>>>>> > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> If you were willing and able to get a CEP together for automated > repair based on the above material, given you've done the work and have the > proof points it's working at scale, I think this would be a _huge > contribution_ to the community. > > > >>>>>>>>> > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote: > > > >>>>>>>>> > > >>>>>>>>>> Is anyone going to file an official CEP for this? > > > >>>>>>>>>> > > >>>>>>>>>> As mentioned in this email thread, here is one of the solution's > [design doc](https://docs.google.com/document/d/1CJWxjEi- > mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0) and source > code on a private Apache Cassandra patch. Could you go through it and let me > know what you think? > > > >>>>>>>>>> > > >>>>>>>>>> > > > >>>>>>>>>> > > >>>>>>>>>> Jaydeep > > > >>>>>>>>>> > > >>>>>>>>>> > > > >>>>>>>>>> > > >>>>>>>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad > <[rustyrazorbl...@apache.org](mailto:rustyrazorbl...@apache.org)> wrote: > > > >>>>>>>>>> > > >>>>>>>>>>> > That said I would happily support an effort to bring repair > scheduling to the sidecar immediately. This has nothing blocking it, and would > potentially enable the sidecar to provide an official repair scheduling > solution that is compatible with current or even previous versions of the > database. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > >>>>>>>>>>> > > >>>>>>>>>>> This is something I hadn't thought much about, and is a pretty > good argument for using the sidecar initially. There's a lot of deployments > out there and having an official repair option would be a big win. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > >>>>>>>>>>> > > >>>>>>>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote: > > > >>>>>>>>>>> > > >>>>>>>>>>> > I agree that it would be ideal for Cassandra to have a repair > scheduler in-DB. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > >>>>>>>>>>> > That said I would happily support an effort to bring repair > scheduling to the sidecar immediately. This has nothing blocking it, and would > potentially enable the sidecar to provide an official repair scheduling > solution that is compatible with current or even previous versions of the > database. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > >>>>>>>>>>> > Once TCM has landed, we’ll have much stronger primitives for > repair orchestration in the database itself. But I don’t think that should > block progress on a repair scheduling solution in the sidecar, and there is > nothing that would prevent someone from continuing to use a sidecar-based > solution in perpetuity if they preferred. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > >>>>>>>>>>> > \- Scott > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > >>>>>>>>>>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad > <[rustyrazorbl...@apache.org](mailto:rustyrazorbl...@apache.org)> wrote: > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > >>>>>>>>>>> > > I'm 100% in favor of repair being part of the core DB, not > the sidecar. The current (and past) state of things where running the DB > correctly *requires* running a separate process (either community maintained > or official C* sidecar) is incredibly painful for folks. The idea that your > data integrity needs to be opt-in has never made sense to me from the > perspective of either the product or the end user. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > >>>>>>>>>>> > > I've worked with way too many teams that have either > configured this incorrectly or not at all. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > >>>>>>>>>>> > > Ideally Cassandra would ship with repair built in and on by > default. Power users can disable if they want to continue to maintain their > own repair tooling for some reason. > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > >>>>>>>>>>> > > Jon > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote: > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> All, > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> We had a brief discussion in [2] about the Uber article [1] > where they talk about having integrated repair into Cassandra and how great > that is. I expressed my disappointment that they didn't work with the > community on that (Uber, if you are listening time to make amends 🙂) and it > turns out Joey already had the idea and wrote the code [3] - so I wanted to > start a discussion to gauge interest and maybe how to revive that effort. > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> Thanks, > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> German > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> [1] <https://www.uber.com/blog/how-uber-optimized-cassandra- > operations-at-scale/> > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> [2] <https://the- > asf.slack.com/archives/CK23JSY2K/p1690225062383619> > > > >>>>>>>>>>> > > >>>>>>>>>>> > >> [3] <https://issues.apache.org/jira/browse/CASSANDRA-14346> > > > >>>>>>>>>>> > > >>>>>>>>>>> > > > > >>>>>>>>> > > >>>>>>>>> > > > >>>>>> > > >>>>>> > > > >
