Re: Increase compaction performance

2016-05-20 Thread Fabrice Facorat
@Alain: Indeed when repairing (or bootstraping) all sstables end up in L0 as original level is not passed down to the node. So cassandra end up compacting a lot of sstables in L0 before trying to make them move to upper levels. The issue still exist in 2.1 and is even worse as you have less concu

Re: Increase compaction performance

2016-03-10 Thread Alain RODRIGUEZ
Hi Michal, Sorry about the delay answering here. The value you gave (10) looks a lot like what I had to do in the past in the cluster I managed. I described the issue here: https://issues.apache.org/jira/browse/CASSANDRA-9509 A few people hit this issue already. Hope you were able to successfull

Re: Increase compaction performance

2016-03-04 Thread Fabrice Facorat
Any news on this ? We also have issues during repairs when using many LCS tables. We end up with 8k sstables, many pending tasks and dropped mutations We are using Cassandra 2.0.10, on a 24 cores server, with multithreaded compactions enabled. ~$ nodetool getstreamthroughput Current stream throu

Re: Increase compaction performance

2016-02-12 Thread Michał Łowicki
I had to decrease streaming throughput to 10 (from default 200) in order to avoid effect or rising number of SSTables and number of compaction tasks while running repair. It's working very slow but it's stable and doesn't hurt the whole cluster. Will try to adjust configuration gradually to see if

Re: Increase compaction performance

2016-02-11 Thread Michał Łowicki
On Thu, Feb 11, 2016 at 5:38 PM, Alain RODRIGUEZ wrote: > Also, are you using incremental repairs (not sure about the available > options in Spotify Reaper) what command did you run ? > > No. > 2016-02-11 17:33 GMT+01:00 Alain RODRIGUEZ : > >> CPU load is fine, SSD disks below 30% utilization,

Re: Increase compaction performance

2016-02-11 Thread Alain RODRIGUEZ
Also, are you using incremental repairs (not sure about the available options in Spotify Reaper) what command did you run ? 2016-02-11 17:33 GMT+01:00 Alain RODRIGUEZ : > CPU load is fine, SSD disks below 30% utilization, no long GC pauses > > > > What is your current compaction throughput ? The

Re: Increase compaction performance

2016-02-11 Thread Alain RODRIGUEZ
> > CPU load is fine, SSD disks below 30% utilization, no long GC pauses What is your current compaction throughput ? The current value of 'concurrent_compactors' (cassandra.yaml or through JMX) ? nodetool getcompactionthroughput How to speed up compaction? Increased compaction throughput and

Increase compaction performance

2016-02-11 Thread Michał Łowicki
Hi, Using 2.1.12 across 3 DCs. Each DC has 8 nodes. Trying to run repair using Cassandra Reaper but nodes after couple of hours are full of pending compaction tasks (regular not the ones about validation) CPU load is fine, SSD disks below 30% utilization, no long GC pauses. How to speed up compa