Re: Proposing an Apache Cassandra Management process
I have a few clarifications - The scope of the management process is not to simply run repair scheduling. Repair scheduling is one of the many features we could implement or adopt from existing sources. So could we please split the Management Process discussion and the repair scheduling? After re-reading the management process proposal, I see we missed to communicate a basic idea in the document. We wanted to take a pluggable approach to various activities that the management process could perform. This could accommodate different implementations of common activities such as repair. The management process would provide the basic framework and it would have default implementations for some of the basic activities. This would allow for speedier iteration cycles and keep things extensible. Turning to some questions that Jon and others have raised, when I +1, my intention is to fully contribute and stay with this community. That said, things feel rushed for some but for me it feels like analysis paralysis. We're looking for actionable feedback and to discuss the management process _not_ repair scheduling solutions. Thanks, Dinesh On Sep 12, 2018, at 6:24 PM, sankalp kohli wrote: Here is a list of open discussion points from the voting thread. I think some are already answered but I will still gather these questions here. >From several people: 1. Vote is rushed and we need more time for discussion. >From Sylvain 2. About the voting process...I think that was addressed by Jeff Jirsa and deserves a separate thread as it is not directly related to this thread. 3. Does the project need a side car. >From Jonathan Haddad 4. Are people doing +1 willing to contribute >From Jonathan Ellis 5. List of feature set, maturity, maintainer availability from Reaper or any other project being donated. Mick Semb Wever 6. We should not vote on these things and instead build consensus. Open Questions from this thread 7. What technical debts we are talking about in Reaper. Can someone give concrete examples. 8. What is the timeline of donating Reaper to Apache Cassandra. On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli wrote: (Using this thread and not the vote thread intentionally) For folks talking about vote being rushed. I would use the email from Joseph to show this is not rushed. There was no email on this thread for 4 months until I pinged. Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to come up with design goals for a repair scheduler that could work at Netflix scale. ~Feb 2017: Netflix believes that the fundamental design gaps prevented us from using Reaper as it relies heavily on remote JMX connections and central coordination. Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available and distributed repair scheduling sidecar/tool. He is encouraged by multiple committers to build repair scheduling into the daemon itself and not as a sidecar so the database is truly eventually consistent. ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at NGCC, Vinay and myself prototype the distributed repair scheduler within Priam and roll it out at Netflix scale. Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page design document for adding repair scheduling to the daemon itself and open the design up for feedback from the community. We get feedback from Alex, Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals to contribute Reaper at this point. We hear the consensus that the community would prefer repair scheduling in a separate distributed sidecar rather than in the daemon itself and we re-work the design to match this consensus, re-aligning with our original proposal at NGCC. Apr 2018: Blake brings the discussion of repair scheduling to the dev list ( https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E ). Many community members give positive feedback that we should solve it as part of Cassandra and there is still no mention of contributing Reaper at this point. The last message is my attempted summary giving context on how we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and ship them with Cassandra. Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document for gathering feedback on a general management sidecar. Sankalp and Dinesh encourage Vinay and myself to kickstart that sidecar using the repair scheduler patch Apr 2018: Dinesh reaches out to the dev list ( https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E ) about the general management process to gain further feedback. All feedback remains positive as it is a potential place for multiple community members to contribute their various sidecar functionality. May-Jul 2017: Vinay and I work on creating a basic sidecar for running the repair scheduler bas
Re: Proposing an Apache Cassandra Management process
Most of the discussion and work was done off the mailing list - there's a big risk involved when folks disappear for months at a time and resurface with big pile of code plus an agenda that you failed to loop everyone in on. In addition, by your own words the design document didn't accurately describe what was being built. I don't write this to try to argue about it, I just want to put some perspective for those of us that weren't part of this discussion on a weekly basis over the last several months. Going forward let's keep things on the ML so we can avoid confusion and frustration for all parties. With that said - I think Blake made a really good point here and it's helped me understand the scope of what's being built better. Looking at it from a different perspective it doesn't seem like there's as much overlap as I had initially thought. There's the machinery that runs certain tasks (what Joey has been working on) and the user facing side of exposing that information in management tool. I do appreciate (and like) the idea of not trying to boil the ocean, and working on things incrementally. Putting a thin layer on top of Cassandra that can perform cluster wide tasks does give us an opportunity to move in the direction of a general purpose user-facing admin tool without committing to trying to write the full stack all at once (or even make decisions on it now). We do need a sensible way of doing rolling restarts / scrubs / scheduling and Reaper wasn't built for that, and even though we can add it I'm not sure if it's the best mechanism for the long term. So if your goal is to add maturity to the project by making cluster wide tasks easier by providing a framework to build on top of, I'm in favor of that and I don't see it as antithetical to what I had in mind with Reaper. Rather, the two are more complementary than I had originally realized. Jon On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID wrote: > I have a few clarifications - > The scope of the management process is not to simply run repair > scheduling. Repair scheduling is one of the many features we could > implement or adopt from existing sources. So could we please split the > Management Process discussion and the repair scheduling? > After re-reading the management process proposal, I see we missed to > communicate a basic idea in the document. We wanted to take a pluggable > approach to various activities that the management process could perform. > This could accommodate different implementations of common activities such > as repair. The management process would provide the basic framework and it > would have default implementations for some of the basic activities. This > would allow for speedier iteration cycles and keep things extensible. > Turning to some questions that Jon and others have raised, when I +1, my > intention is to fully contribute and stay with this community. That said, > things feel rushed for some but for me it feels like analysis paralysis. > We're looking for actionable feedback and to discuss the management process > _not_ repair scheduling solutions. > Thanks, > Dinesh > > > > On Sep 12, 2018, at 6:24 PM, sankalp kohli wrote: > Here is a list of open discussion points from the voting thread. I think > some are already answered but I will still gather these questions here. > > From several people: > 1. Vote is rushed and we need more time for discussion. > > From Sylvain > 2. About the voting process...I think that was addressed by Jeff Jirsa and > deserves a separate thread as it is not directly related to this thread. > 3. Does the project need a side car. > > From Jonathan Haddad > 4. Are people doing +1 willing to contribute > > From Jonathan Ellis > 5. List of feature set, maturity, maintainer availability from Reaper or > any other project being donated. > > Mick Semb Wever > 6. We should not vote on these things and instead build consensus. > > Open Questions from this thread > 7. What technical debts we are talking about in Reaper. Can someone give > concrete examples. > 8. What is the timeline of donating Reaper to Apache Cassandra. > > On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli > wrote: > > > (Using this thread and not the vote thread intentionally) > For folks talking about vote being rushed. I would use the email from > Joseph to show this is not rushed. There was no email on this thread for 4 > months until I pinged. > > > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to > come up with design goals for a repair scheduler that could work at Netflix > scale. > > ~Feb 2017: Netflix believes that the fundamental design gaps prevented us > from using Reaper as it relies heavily on remote JMX connections and > central coordination. > > Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available > and distributed repair scheduling sidecar/tool. He is encouraged by > multiple committers to build repair scheduling into the daemon itse
Re: Proposing an Apache Cassandra Management process
I will update the document to add that point. The document did not mean to serve as a design or architectural document but rather something that would spark a discussion on the idea. Dinesh On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad wrote: Most of the discussion and work was done off the mailing list - there's a big risk involved when folks disappear for months at a time and resurface with big pile of code plus an agenda that you failed to loop everyone in on. In addition, by your own words the design document didn't accurately describe what was being built. I don't write this to try to argue about it, I just want to put some perspective for those of us that weren't part of this discussion on a weekly basis over the last several months. Going forward let's keep things on the ML so we can avoid confusion and frustration for all parties. With that said - I think Blake made a really good point here and it's helped me understand the scope of what's being built better. Looking at it from a different perspective it doesn't seem like there's as much overlap as I had initially thought. There's the machinery that runs certain tasks (what Joey has been working on) and the user facing side of exposing that information in management tool. I do appreciate (and like) the idea of not trying to boil the ocean, and working on things incrementally. Putting a thin layer on top of Cassandra that can perform cluster wide tasks does give us an opportunity to move in the direction of a general purpose user-facing admin tool without committing to trying to write the full stack all at once (or even make decisions on it now). We do need a sensible way of doing rolling restarts / scrubs / scheduling and Reaper wasn't built for that, and even though we can add it I'm not sure if it's the best mechanism for the long term. So if your goal is to add maturity to the project by making cluster wide tasks easier by providing a framework to build on top of, I'm in favor of that and I don't see it as antithetical to what I had in mind with Reaper. Rather, the two are more complementary than I had originally realized. Jon On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID wrote: > I have a few clarifications - > The scope of the management process is not to simply run repair > scheduling. Repair scheduling is one of the many features we could > implement or adopt from existing sources. So could we please split the > Management Process discussion and the repair scheduling? > After re-reading the management process proposal, I see we missed to > communicate a basic idea in the document. We wanted to take a pluggable > approach to various activities that the management process could perform. > This could accommodate different implementations of common activities such > as repair. The management process would provide the basic framework and it > would have default implementations for some of the basic activities. This > would allow for speedier iteration cycles and keep things extensible. > Turning to some questions that Jon and others have raised, when I +1, my > intention is to fully contribute and stay with this community. That said, > things feel rushed for some but for me it feels like analysis paralysis. > We're looking for actionable feedback and to discuss the management process > _not_ repair scheduling solutions. > Thanks, > Dinesh > > > > On Sep 12, 2018, at 6:24 PM, sankalp kohli wrote: > Here is a list of open discussion points from the voting thread. I think > some are already answered but I will still gather these questions here. > > From several people: > 1. Vote is rushed and we need more time for discussion. > > From Sylvain > 2. About the voting process...I think that was addressed by Jeff Jirsa and > deserves a separate thread as it is not directly related to this thread. > 3. Does the project need a side car. > > From Jonathan Haddad > 4. Are people doing +1 willing to contribute > > From Jonathan Ellis > 5. List of feature set, maturity, maintainer availability from Reaper or > any other project being donated. > > Mick Semb Wever > 6. We should not vote on these things and instead build consensus. > > Open Questions from this thread > 7. What technical debts we are talking about in Reaper. Can someone give > concrete examples. > 8. What is the timeline of donating Reaper to Apache Cassandra. > > On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli > wrote: > > > (Using this thread and not the vote thread intentionally) > For folks talking about vote being rushed. I would use the email from > Joseph to show this is not rushed. There was no email on this thread for 4 > months until I pinged. > > > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to > come up with design goals for a repair scheduler that could work at Netflix > scale. > > ~Feb 2017: Netflix believes that the fundamental design gaps prevented us > from using Reaper as it relies
partitioning and CRC
Hi all, I am trying to understand where exactly digests and checksums are being used in Cassandra. In my understanding, Murmur3 hashing is used with murmur3 partitioning scheme which is also the default configuration. CRC32 is used for data corruption and repair. MD5 is used for partitioning only when random partitioning is configured. Can you please correct if I'm missing something here? Thanks, Preetika
