What you’re describing seems very close to what’s discussed in https://issues.apache.org/jira/browse/CASSANDRA-10979 - worth reading that ticket a bit.
There does seem to be a check for STCS in L0 before it tries higher levels: https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java#L324-L326 Why it’s doing that within the for loop (https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java#L310 ) is unexpected to me, though – Carl / Marcus, any insight into why it’s within the loop instead of before it? From: Marcus Olsson <marcus.ols...@ericsson.com> Organization: Ericsson AB Reply-To: "dev@cassandra.apache.org" <dev@cassandra.apache.org> Date: Wednesday, November 23, 2016 at 7:52 AM To: "dev@cassandra.apache.org" <dev@cassandra.apache.org> Subject: STCS in L0 behaviour Hi everyone, TL;DR Should LCS be changed to always prefer an STCS compaction in L0 if it's falling behind? Assuming that STCS in L0 is enabled. Currently LCS seems to check if there is a possible L0->L1 compaction before checking if it's falling behind, which in our case used between 15-30% of the compaction thread CPU. TL;DR So first some background: We have a Apache Cassandra 2.2 cluster running with a high load. In that cluster there is a table with a moderate amount of writes per second that is using LeveledCompactionStrategy. The test was to run repair on that table while we monitored the cluster through JMC and with Flight Recordings enabled. This resulted in a large amount of sstables for that table, which I assume others have experienced as well. In this case I think it was between 15-20k. >From the Flight Recording one thing we saw was that 15-30% of the CPU time in >each of the compaction threads was spent on "getNextBackgroundTask()" which >retrieves the next compaction job. With some further investigation this seems >to mostly be when it's checking for overlap in L0 sstables before performing >an L0->L1 compaction. There is a JIRA which seems to be related to this >https://issues.apache.org/jira/browse/CASSANDRA-11571 which we backported to >2.2 and tested. In our testing it seemed to improve the situation but it was >still using noticeable CPU. My interpretation of the current logic of LCS is (if STCS in L0 is enabled): 1. Check each level (L1+) - If a L1+ compaction is needed check if L0 is behind and do STCS if that's the case, otherwise do the L1+ compaction. 2. Check L0 -> L1 compactions and if none is needed/possible check for STCS in L0. My proposal is to change this behavior to always check if L0 is far behind first and do a STCS compaction in that case. This would avoid the overlap check for L0 -> L1 compactions when L0 is behind and I think it makes sense since we already prefer STCS to L1+ compactions. This would not solve the repair situation, but it would lower some of the impact that repair has on LCS. For what version this could get in I think trunk would be enough since compaction is pluggable. -- MARCUS OLSSON Software Developer Ericsson Sweden marcus.ols...@ericsson.com www.ericsson.com
smime.p7s
Description: S/MIME cryptographic signature
