Re: constant CMS GC using CPU time

2012-10-26 Thread aaron morton
> How does compaction_throughput relate to memory usage? It reduces the rate of memory allocation. e.g. Say normally ParNew can keep up with the rate of memory usage without stopping for too long: so the rate of promotion is low'ish and every thing is allocated to Eden. If the allocation rate

Re: constant CMS GC using CPU time

2012-10-25 Thread Bryan Talbot
On Thu, Oct 25, 2012 at 4:15 AM, aaron morton wrote: > This sounds very much like "my heap is so consumed by (mostly) bloom >> filters that I am in steady state GC thrash." >> > > Yes, I think that was at least part of the issue. > > > The rough numbers I've used to estimate working set are: > >

Re: constant CMS GC using CPU time

2012-10-25 Thread aaron morton
> This sounds very much like "my heap is so consumed by (mostly) bloom > filters that I am in steady state GC thrash." > > Yes, I think that was at least part of the issue. The rough numbers I've used to estimate working set are: * bloom filter size for 400M rows at 0.00074 fp without java fudge

Re: constant CMS GC using CPU time

2012-10-25 Thread aaron morton
> Regarding memory usage after a repair ... Are the merkle trees kept around? > They should not be. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 24/10/2012, at 4:51 PM, B. Todd Burruss wrote: > Regarding memory usage after a repair .

Re: constant CMS GC using CPU time

2012-10-24 Thread Bryan Talbot
On Wed, Oct 24, 2012 at 2:38 PM, Rob Coli wrote: > On Mon, Oct 22, 2012 at 8:38 AM, Bryan Talbot > wrote: > > The nodes with the most data used the most memory. All nodes are > affected > > eventually not just one. The GC was on-going even when the nodes were > not > > compacting or running a

Re: constant CMS GC using CPU time

2012-10-24 Thread Rob Coli
On Mon, Oct 22, 2012 at 8:38 AM, Bryan Talbot wrote: > The nodes with the most data used the most memory. All nodes are affected > eventually not just one. The GC was on-going even when the nodes were not > compacting or running a heavy application load -- even when the main app was > paused con

Re: constant CMS GC using CPU time

2012-10-23 Thread B. Todd Burruss
Regarding memory usage after a repair ... Are the merkle trees kept around? On Oct 23, 2012 3:00 PM, "Bryan Talbot" wrote: > On Mon, Oct 22, 2012 at 6:05 PM, aaron morton wrote: > >> The GC was on-going even when the nodes were not compacting or running a >> heavy application load -- even when th

Re: constant CMS GC using CPU time

2012-10-23 Thread Bryan Talbot
On Mon, Oct 22, 2012 at 6:05 PM, aaron morton wrote: > The GC was on-going even when the nodes were not compacting or running a > heavy application load -- even when the main app was paused constant the GC > continued. > > If you restart a node is the onset of GC activity correlated to some event?

Re: constant CMS GC using CPU time

2012-10-23 Thread Bryan Talbot
These GC settings are the default (recommended?) settings from cassandra-env. I added the UseCompressedOops. -Bryan On Mon, Oct 22, 2012 at 6:15 PM, Will @ SOHO wrote: > On 10/22/2012 09:05 PM, aaron morton wrote: > > # GC tuning options > JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" > JVM_OPTS="$J

Re: constant CMS GC using CPU time

2012-10-22 Thread Will @ SOHO
On 10/22/2012 09:05 PM, aaron morton wrote: # GC tuning options JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC" JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled" JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8" JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1" JVM_OPTS="

Re: constant CMS GC using CPU time

2012-10-22 Thread aaron morton
> The GC was on-going even when the nodes were not compacting or running a > heavy application load -- even when the main app was paused constant the GC > continued. If you restart a node is the onset of GC activity correlated to some event? > As a test we dropped the largest CF and the memory

Re: constant CMS GC using CPU time

2012-10-22 Thread Bryan Talbot
The memory usage was correlated with the size of the data set. The nodes were a bit unbalanced which is normal due to variations in compactions. The nodes with the most data used the most memory. All nodes are affected eventually not just one. The GC was on-going even when the nodes were not co

Re: constant CMS GC using CPU time

2012-10-22 Thread aaron morton
If you are using the default settings I would try to correlate the GC activity with some application activity before tweaking. If this is happening on one machine out of 4 ensure that client load is distributed evenly. See if the raise in GC activity us related to Compaction, repair or an incr

Re: constant CMS GC using CPU time

2012-10-21 Thread Radim Kolar
Dne 18.10.2012 20:06, Bryan Talbot napsal(a): In a 4 node cluster running Cassandra 1.1.5 with sun jvm 1.6.0_29-b11 (64-bit), the nodes are often getting "stuck" in state where CMS collections of the old space are constantly running. you need more java heap memory

Re: constant CMS GC using CPU time

2012-10-19 Thread Bryan Talbot
ok, let me try asking the question a different way ... How does cassandra use memory and how can I plan how much is needed? I have a 1 GB memtable and 5 GB total heap and that's still not enough even though the number of concurrent connections and garbage generation rate is fairly low. If I were

constant CMS GC using CPU time

2012-10-18 Thread Bryan Talbot
In a 4 node cluster running Cassandra 1.1.5 with sun jvm 1.6.0_29-b11 (64-bit), the nodes are often getting "stuck" in state where CMS collections of the old space are constantly running. The JVM configuration is using the standard settings in cassandra-env -- relevant settings are included below.