Re: New contribution - Burst Hour Compaction Strategy
Hi I've addressed the issues with Git. I believe this is what Stefan asking for: https://github.com/sedulam/cassandra/tree/12201 I've also added more tests for BHCS, including more for wide rows following Jeff's suggestion. Thanks for the directions so far! If there's something else you would like to see tested or some metrics, please let me know what would be relevant. All the best Pedro Gordo On 13 June 2017 at 15:43, Pedro Gordo wrote: > Hi all > > Although a couple of people engaged with me directly to talk about BHCS, I > would also like to get the community opinion on this, so I thought I could > get the discussion started by saying what the advantages would be and in > which type of tables BHCS would do a good job. Please keep in mind that all > my assumptions are without any real world experience on Cassandra, so this > is where I expect to see some input of the C* veterans to help me steer > BHCS implementation in the right direction if needed. This is a long email, > so there's a TLDR if you don't want to read everything. This is intended > for high-level discussion. For code level discussion, please refer to the > document in JIRA. > I'm aware that some might not like that no compaction occurs outside of > the burst hour, but I thought of solutions for that, so please read the > planned improvements below. > > *TL;DR* > BHCS tries to address these issues with the current compaction strategies: > - Necessity of allocating large storage during big compactions in STCS -> > Through the sstable_max_size property of BHCS, we can keep SSTables below a > certain size, so we wouldn't have issues with size during compaction > - We might get to a point where to return the results of a query, we need > to read from a large number of SSTables -> BHCS addresses this by making > sure that the number of SSTables where a key exists will be consistently > maintained at a low level after every compaction. The number of SSTables > where a key exists is configurable, so in the limit, you could set it to 1 > for optimal read performance. > - Continuous high I/O of LCS -> addressed by the scheduling feature of > BHCS. > > *Longer explanation:* > > *Where would it be advantageous using BHCS?* > - Read-bound tables: due to BHCS maintaining the number of key copies at a > low level, the read speed would be consistently fast. Since there's not a > lot of writes in this type of table, even if there are new SSTables > produced containing that key, the number SSTables containing that key would > be set again to 1 after burst hour (BH). > - Write-bound tables: in this scenario, there's a lot of SSTables created > outside of BH, but few reads, so the issue with existing strategies would > be a continuous high I/O dedicated to compaction. With BHCS during these > active hours, we would have an increase in disk size, but I assume that > this disk increase outside the BH would be tolerable since a lot of space > would be released during the burst. Still, if that's a big issue, I plan to > address this with the improvement (1). > > *Where is BHCS NOT recommended and what improvements can be done to make > it viable?* > - Read and write-heavy tables: because outside BH, SSTables would increase > until the burst kicks in, there can be an increase in the read speed and > disk used space. This could also be solved with improvement (1), (3) or (5). > > *Planned Improvements:* > (1) - The user could indicate that he wants continuous compaction. This > would change the strategy in such a way that outside of the Burst Hour, > STCS would be used to maintain an acceptable read speed and disk used > space. And then when BH would kick in, it would set key copies and disk > size again to optimal levels. > (2) - During table creation the user, might not be aware of the compaction > configurable details, so a user-friendly configuration would be provided. > If the user sets the table as a Write-and-Read heavy table, then > improvement (1) would be activated. Otherwise, the strategy would default > to its current config to save resources during the outside the BH. > (3) - Instead, of just one burst hour, we could set several periods for > BHCS to run during the day (for instance, every 3 hours or another > schedule). > > *Ideas:* > (4) - Continuously evaluate how many pending compactions we have and I/O > status, and then based on that, we start (or not) the compaction. > (5) - If outside the BH, the size for all the SSTables in a family set > reaches a certain threshold, then background compaction can occur anyway. > This threshold should be elevated due to the high CPU usage of BHCS. > > Please let me know your thoughts on this. Thanks! > > Best regards > > Pedro Gordo > > On 10 June 2017 at 22:22, J. D. Jordan wrote: > >> GitHub has some good guides on how to use git and make a pull request for >> a project. >> >> https://guides.github.com/introduction/flow/ >> https://guides.github.com/activities/forking/ >> >> On Jun 10, 2017, at 3:
Re: New contribution - Burst Hour Compaction Strategy
The main issue I see with this is "Read all the SSTables and detect which partition keys are present in more than the compaction minimum threshold value" . This is quite expensive and will be using quite a lot of I/O to calculate. What makes writing a compaction strategy so difficult is calculating what compactions to do without spending lots of I/O. I would like to see benchmarks of this compaction strategy and where it offers benefits to the existing compaction strategies. In particular you state one of the motivations is to "Continuous high I/O of LCS -> addressed by the scheduling feature of BHCS" however I don't see how you have achieved this with the read I/O required to calculate which sstables contain a key. On 15 June 2017 at 07:49, Pedro Gordo wrote: > Hi > > I've addressed the issues with Git. I believe this is what Stefan asking > for: https://github.com/sedulam/cassandra/tree/12201 > I've also added more tests for BHCS, including more for wide rows following > Jeff's suggestion. > > Thanks for the directions so far! If there's something else you would like > to see tested or some metrics, please let me know what would be relevant. > > All the best > > > Pedro Gordo > > On 13 June 2017 at 15:43, Pedro Gordo wrote: > > > Hi all > > > > Although a couple of people engaged with me directly to talk about BHCS, > I > > would also like to get the community opinion on this, so I thought I > could > > get the discussion started by saying what the advantages would be and in > > which type of tables BHCS would do a good job. Please keep in mind that > all > > my assumptions are without any real world experience on Cassandra, so > this > > is where I expect to see some input of the C* veterans to help me steer > > BHCS implementation in the right direction if needed. This is a long > email, > > so there's a TLDR if you don't want to read everything. This is intended > > for high-level discussion. For code level discussion, please refer to the > > document in JIRA. > > I'm aware that some might not like that no compaction occurs outside of > > the burst hour, but I thought of solutions for that, so please read the > > planned improvements below. > > > > *TL;DR* > > BHCS tries to address these issues with the current compaction > strategies: > > - Necessity of allocating large storage during big compactions in STCS -> > > Through the sstable_max_size property of BHCS, we can keep SSTables > below a > > certain size, so we wouldn't have issues with size during compaction > > - We might get to a point where to return the results of a query, we need > > to read from a large number of SSTables -> BHCS addresses this by making > > sure that the number of SSTables where a key exists will be consistently > > maintained at a low level after every compaction. The number of SSTables > > where a key exists is configurable, so in the limit, you could set it to > 1 > > for optimal read performance. > > - Continuous high I/O of LCS -> addressed by the scheduling feature of > > BHCS. > > > > *Longer explanation:* > > > > *Where would it be advantageous using BHCS?* > > - Read-bound tables: due to BHCS maintaining the number of key copies at > a > > low level, the read speed would be consistently fast. Since there's not a > > lot of writes in this type of table, even if there are new SSTables > > produced containing that key, the number SSTables containing that key > would > > be set again to 1 after burst hour (BH). > > - Write-bound tables: in this scenario, there's a lot of SSTables created > > outside of BH, but few reads, so the issue with existing strategies would > > be a continuous high I/O dedicated to compaction. With BHCS during these > > active hours, we would have an increase in disk size, but I assume that > > this disk increase outside the BH would be tolerable since a lot of space > > would be released during the burst. Still, if that's a big issue, I plan > to > > address this with the improvement (1). > > > > *Where is BHCS NOT recommended and what improvements can be done to make > > it viable?* > > - Read and write-heavy tables: because outside BH, SSTables would > increase > > until the burst kicks in, there can be an increase in the read speed and > > disk used space. This could also be solved with improvement (1), (3) or > (5). > > > > *Planned Improvements:* > > (1) - The user could indicate that he wants continuous compaction. This > > would change the strategy in such a way that outside of the Burst Hour, > > STCS would be used to maintain an acceptable read speed and disk used > > space. And then when BH would kick in, it would set key copies and disk > > size again to optimal levels. > > (2) - During table creation the user, might not be aware of the > compaction > > configurable details, so a user-friendly configuration would be provided. > > If the user sets the table as a Write-and-Read heavy table, then > > improvement (1) would be activated. Otherwise, the strategy would default > > to
Re: New contribution - Burst Hour Compaction Strategy
Hi Pedro, I did a quick read through of your strategy, and have a few personal thoughts: First, writing a compaction strategy is a lot of work, and it's great to see new contributors take on ambitious projects. There are even a handful of ideas in here that may be useful to other strategies. The overall concept is interesting - many companies have "peak" times and "offpeak" times, and being able to run compaction only during offpeak may be really helpful. This concept is actually in the old wiki dating back many years , for example Ed Capriolo gave a talk ( https://www.slideshare.net/edwardcapriolo/m6d-cassandrapresentation - check out slide #28 ) where he showed how to achieve this with cron and nodetool. The actual logic you use to select candidates probably isn't perfect, because it can be pretty nuanced. But rather than focus on that, if we take advantage of the larger concept that it's useful to be able to turn compaction on/off on a schedule, there may be another opportunity - rather than try to re-implement some of the concepts of LCS without using LCS, you could just make BurstHourCompactionStrategy a wrapper around any user-specified compaction strategy. That is, it may be much less work and much more valuable if you actually let users specify which underlying compaction strategy to wrap, and then simply use the underlying the wrapped getNextBackgroundCompactionTask() For the project to be willing to accept a 5th compaction strategy, I imagine committers will want to see some benchmarks and hopefully some concrete examples of how it's beneficial and solves a problem that can't be better solved in other ways. I think at a high level many people can understand how it's useful, but you may want to compare/contrast it to Ed's method in the deck above (in particular, using nodetool and cron, you can have multiple "compaction-enabled" windows during the day, and you can throttle it so that it never fully stops, but slows down enough that it doesn't impact latencies. Again, that's just my personal thought based on a quick read through. Nice work so far! - Jeff On Wed, Jun 14, 2017 at 2:49 PM, Pedro Gordo wrote: > Hi > > I've addressed the issues with Git. I believe this is what Stefan asking > for: https://github.com/sedulam/cassandra/tree/12201 > I've also added more tests for BHCS, including more for wide rows following > Jeff's suggestion. > > Thanks for the directions so far! If there's something else you would like > to see tested or some metrics, please let me know what would be relevant. > > All the best > > > Pedro Gordo > > On 13 June 2017 at 15:43, Pedro Gordo wrote: > > > Hi all > > > > Although a couple of people engaged with me directly to talk about BHCS, > I > > would also like to get the community opinion on this, so I thought I > could > > get the discussion started by saying what the advantages would be and in > > which type of tables BHCS would do a good job. Please keep in mind that > all > > my assumptions are without any real world experience on Cassandra, so > this > > is where I expect to see some input of the C* veterans to help me steer > > BHCS implementation in the right direction if needed. This is a long > email, > > so there's a TLDR if you don't want to read everything. This is intended > > for high-level discussion. For code level discussion, please refer to the > > document in JIRA. > > I'm aware that some might not like that no compaction occurs outside of > > the burst hour, but I thought of solutions for that, so please read the > > planned improvements below. > > > > *TL;DR* > > BHCS tries to address these issues with the current compaction > strategies: > > - Necessity of allocating large storage during big compactions in STCS -> > > Through the sstable_max_size property of BHCS, we can keep SSTables > below a > > certain size, so we wouldn't have issues with size during compaction > > - We might get to a point where to return the results of a query, we need > > to read from a large number of SSTables -> BHCS addresses this by making > > sure that the number of SSTables where a key exists will be consistently > > maintained at a low level after every compaction. The number of SSTables > > where a key exists is configurable, so in the limit, you could set it to > 1 > > for optimal read performance. > > - Continuous high I/O of LCS -> addressed by the scheduling feature of > > BHCS. > > > > *Longer explanation:* > > > > *Where would it be advantageous using BHCS?* > > - Read-bound tables: due to BHCS maintaining the number of key copies at > a > > low level, the read speed would be consistently fast. Since there's not a > > lot of writes in this type of table, even if there are new SSTables > > produced containing that key, the number SSTables containing that key > would > > be set again to 1 after burst hour (BH). > > - Write-bound tables: in this scenario, there's a lot of SSTables created > > outside of BH, but few reads, so the issue with existing stra
