Re: TWCS generates large numbers of sstables on only some nodes

2019-07-16 Thread Jeff Jirsa
On Tue, Jul 16, 2019 at 8:54 AM Carl Mueller wrote: > stays consistently in the 40-60 range, but only recent tables are being > compacted. > > What I fear is that TWCS when it hits a certain compaction threshold keeps > compacting the same tables adding a slice of the most recently flushed data >

Re: TWCS generates large numbers of sstables on only some nodes

2019-07-16 Thread Oleksandr Shulgin
On Tue, Jul 16, 2019 at 5:54 PM Carl Mueller wrote: > stays consistently in the 40-60 range, but only recent tables are being > compacted. > I would be alarmed at this point. It definitely feels like not aggressive enough compaction: can you relax the throttling or afford to have more concurren

Re: TWCS generates large numbers of sstables on only some nodes

2019-07-16 Thread Carl Mueller
stays consistently in the 40-60 range, but only recent tables are being compacted. What I fear is that TWCS when it hits a certain compaction threshold keeps compacting the same tables adding a slice of the most recently flushed data and falls behind. I'd rather it compacted fragments of sstable

Re: TWCS generates large numbers of sstables on only some nodes

2019-07-15 Thread Oleksandr Shulgin
On Mon, Jul 15, 2019 at 6:20 PM Carl Mueller wrote: > Related to our overstreaming, we have a cluster of about 25 nodes, with > most at about 1000 sstable files (Data + others). > > And about four that are at 20,000 - 30,000 sstable files (Data+Index+etc). > > We have vertically scaled the outlie

TWCS generates large numbers of sstables on only some nodes

2019-07-15 Thread Carl Mueller
Related to our overstreaming, we have a cluster of about 25 nodes, with most at about 1000 sstable files (Data + others). And about four that are at 20,000 - 30,000 sstable files (Data+Index+etc). We have vertically scaled the outlier machines and turned off compaction throttling thinking it was