Re: Making CommitLog pluggable
Hi, I submitted patches for the pluggable CommitLog. Appreciate any comments. Thanks! https://issues.apache.org/jira/browse/CASSANDRA-14062 Rei Odaira 2017-11-06 17:14 GMT-06:00 大平怜 : > Thanks for the feedback, Ariel, > > Based on your comments, we are revisiting our code changes, > and then we will submit a patch for review. > I hope this effort will help further modularize Cassandra > for better maintainability. > > > Thanks, > Rei Odaira > > > 2017-11-06 8:20 GMT-06:00 Ariel Weisberg : > >> Hi, >> >> OK sorry I am very late to the discussion. I think the existing >> consensus around doing it is fine I just think you will find that making >> the commit log pluggable might be a little trickier than making a cache >> which is a glorified K/V store pluggable. >> >> The commit log reaches into a bunch of other internal API during replay >> and even a few at runtime. I think it's a refactor away from >> abstracting out those concerns from the concerns of making log records >> durable, providing notifications of durability, and then releasing them >> though. >> >> If I'm the only person unhappy about breaking the plugins in bug fix >> releases and the resulting problems that creates for anyone who wants to >> operate this hardware in production then I am willing to look past it. >> We could also address it via documentation the plugin link page as well >> as at landing sites for individual plugins. >> >> Otherwise I think this is a bigger commitment from us to a larger API >> then originally scoped and one that restricts what changes can be made >> in bug fix releases to the existing C* commit log. Unless we have >> current version and version next of the plugin API so that we can move >> ahead in bug fix releases without breaking existing plugins. >> >> Thanks, >> Ariel >> >> On Thu, Nov 2, 2017, at 04:46 PM, 大平怜 wrote: >> > Hell Ariel, >> > >> > About the pluggability, we have discussed this topic in the dev list >> last >> > May: >> > https://www.mail-archive.com/dev@cassandra.apache.org/msg11102.html >> > >> > I don't think the whole community has reached a consensus, but >> > the result of the discussion at that time was that >> > 1) We were going to release our extensions as plugins. >> > 2) The project would support the plugin ecosystem by creating a plugin >> > link >> >page on the Web site. >> > >> > Appreciate it if you could shed new light on the discussion. >> > >> > >> > Thanks, >> > Rei Odaira >> > >> > >> > 2017-11-01 17:53 GMT-05:00 Ariel Weisberg : >> > >> > > Hi, >> > > >> > > Just so I don't seem too negative, what I would really like to see is >> an >> > > in tree implementation. The real challenge there is that the hardware >> is >> > > not widely available. If it were something you could get in GCE or AWS >> > > or at least get via an emulator that would be a different story. >> > > >> > > Ariel >> > > >> > > On Wed, Nov 1, 2017, at 06:46 PM, Ariel Weisberg wrote: >> > > > Hi, >> > > > >> > > > OK. It makes sense that most of the existing plumbing is not >> applicable >> > > > since it operates on a filesystem. >> > > > >> > > > How does replay work? Presumably you will need to refactor >> > > > CommitLogReplayer as well? >> > > > >> > > > I think the best way for us to decide whether it's something we >> want in >> > > > tree is to see a patch. You would need to do this even if it doesn't >> > > > make it in tree and you end up having to deploy a patched build. >> > > > >> > > > Pluggability is a little bit of a touchy subject because we don't >> want >> > > > to directly or indirectly become responsible for interfaces to out >> of >> > > > tree implementations. I don't know if there is consensus around >> this, >> > > > but I think even if we made the commit log pluggable it would be >> with >> > > > the understanding that we may change the API even in bug fix >> releases. >> > > > >> > > > Down the line where this becomes tricky is unmaintained out of tree >> > > > implementations that people depend on being broken due to interface >> > > > changes and then no one being around to fix them. People who depend >> on >> > > > the out of tree implementation have no one to complain to but us. >> This >> > > > becomes even more likely when the maintainers aren't using the >> latest >> > > > version of C* and are busy with other things. >> > > > >> > > > You are characterizing the API as being just a few methods on >> CommitLog >> > > > but that isn't true. >> > > > >> > > > These are the imports for CommitLogReplayer >> > > > >> > > > import org.apache.cassandra.concurrent.Stage; >> > > > import org.apache.cassandra.concurrent.StageManager; >> > > > import org.apache.cassandra.config.CFMetaData; >> > > > import org.apache.cassandra.config.Schema; >> > > > import org.apache.cassandra.db.*; >> > > > import org.apache.cassandra.io.util.FastByteArrayInputStream; >> > > > import org.apache.cassandra.io.util.FileUtils; >> > > > import org.apache.cassandra.io.util.RandomAccessReader;
duration based config settings
I ways back I had entire CASSANDRA-13976 out of sheer annoyance to change the hint time to be in minutes instead of ms. Millisecond based resolution is a bit absurd for things like hints. I figured minutes would be better, but after some back and forth realized durations (3h, 30m, etc) would be a lot easier to work with, and would probably be appropriate across the board. I’ve dealt with quite a few clusters in the last year, and I’ve seen a handful of fat fingered config files, or non-standard times that make me bust out a calculator to be sure I’ve got things sorted out right, hence the original issue. Jeff Jirsa suggested migrating to duration types would result in migration pain, and I’m not disagreeing with him. I think we if were to move to duration types, we’d want something like the following: 1. add a blah_blah for every blah_blah_ms setting which accepts a duration 2. convert every setting to use blah_blah 3. if blah_blah_ms is present, use that for blah_blah and set the duration to ms 4. internally everything converts to ms 5. make all node tool commands use durations 6. for ever setting that’s switch to blah_blah, leave a note that says the setting it’s replacing 7. put a warning when people use blah_blah_ms and suggest the conversation to the new config field 8. *sometime* in the future remove _ms. Maybe as far as a year or two down the line. This seems to me like a significant change and I wanted to get some more opinions on the topic before pushing forward. Thoughts? Jon - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: duration based config settings
I'm strongly in favour of it; it's always bugged me, and I hadn't realised it might be contentious to implement. I'd be in favour of never retiring the _ms format though - it's almost free, there's no backward compatibility problems, and it's fairly intuitive so long as we're consistent about it. The only sticking point I can personally see is that there might be a desire to roll this up into a standardisation effort, as we have some duration properties (e.g. key_cache_save_period) which do not have a unit suffix. If we want to standardise on blah_blah_ms, and blah_blah we might want to moonlight those, migrating them to some new blah_blahish_ms and blah_blahish. On 5 December 2017 at 00:24, Jon Haddad wrote: > I ways back I had entire CASSANDRA-13976 out of sheer annoyance to change > the hint time to be in minutes instead of ms. Millisecond based resolution > is a bit absurd for things like hints. I figured minutes would be better, > but after some back and forth realized durations (3h, 30m, etc) would be a > lot easier to work with, and would probably be appropriate across the board. > > I’ve dealt with quite a few clusters in the last year, and I’ve seen a > handful of fat fingered config files, or non-standard times that make me > bust out a calculator to be sure I’ve got things sorted out right, hence > the original issue. > > Jeff Jirsa suggested migrating to duration types would result in migration > pain, and I’m not disagreeing with him. I think we if were to move to > duration types, we’d want something like the following: > > 1. add a blah_blah for every blah_blah_ms setting which accepts a duration > 2. convert every setting to use blah_blah > 3. if blah_blah_ms is present, use that for blah_blah and set the duration > to ms > 4. internally everything converts to ms > 5. make all node tool commands use durations > 6. for ever setting that’s switch to blah_blah, leave a note that says the > setting it’s replacing > 7. put a warning when people use blah_blah_ms and suggest the conversation > to the new config field > 8. *sometime* in the future remove _ms. Maybe as far as a year or two > down the line. > > This seems to me like a significant change and I wanted to get some more > opinions on the topic before pushing forward. Thoughts? > > Jon > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: duration based config settings
To be clear I’m not -1 on this, I just hope we think about upgrading users as we deprecate the old stuff out -- Jeff Jirsa > On Dec 4, 2017, at 4:24 PM, Jon Haddad wrote: > > I ways back I had entire CASSANDRA-13976 out of sheer annoyance to change the > hint time to be in minutes instead of ms. Millisecond based resolution is a > bit absurd for things like hints. I figured minutes would be better, but > after some back and forth realized durations (3h, 30m, etc) would be a lot > easier to work with, and would probably be appropriate across the board. > > I’ve dealt with quite a few clusters in the last year, and I’ve seen a > handful of fat fingered config files, or non-standard times that make me bust > out a calculator to be sure I’ve got things sorted out right, hence the > original issue. > > Jeff Jirsa suggested migrating to duration types would result in migration > pain, and I’m not disagreeing with him. I think we if were to move to > duration types, we’d want something like the following: > > 1. add a blah_blah for every blah_blah_ms setting which accepts a duration > 2. convert every setting to use blah_blah > 3. if blah_blah_ms is present, use that for blah_blah and set the duration to > ms > 4. internally everything converts to ms > 5. make all node tool commands use durations > 6. for ever setting that’s switch to blah_blah, leave a note that says the > setting it’s replacing > 7. put a warning when people use blah_blah_ms and suggest the conversation to > the new config field > 8. *sometime* in the future remove _ms. Maybe as far as a year or two down > the line. > > This seems to me like a significant change and I wanted to get some more > opinions on the topic before pushing forward. Thoughts? > > Jon > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: duration based config settings
> I'd be in favour of never retiring the _ms format though - it's almost > free, there's no backward compatibility problems, and it's fairly intuitive > so long as we're consistent about it. > Agreed on this point. Overall, this will be excellent for usability. - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: duration based config settings
Sure, I’m fine w/ letting the _ms settings work indefinitely. Can revisit retiring them if there’s ever a real need, am OK if that never happens. I’ll create the JIRA. > On Dec 4, 2017, at 5:19 PM, Nate McCall wrote: > >> I'd be in favour of never retiring the _ms format though - it's almost >> free, there's no backward compatibility problems, and it's fairly intuitive >> so long as we're consistent about it. >> > > Agreed on this point. Overall, this will be excellent for usability. > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
