Re: very confused by jmap dump of cassandra

2013-02-22 Thread Hiller, Dean
(Thanks Aaron, I will try those stepsÅ .why are those steps not on this page http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size ) I don't have much disk used at this point(roughly 130G per node). Here are the numbers for the data mount and the commitlog mount(commit log is s

Re: very confused by jmap dump of cassandra

2013-02-21 Thread Mohit Anchlia
Roughly how much data do you have per node? Sent from my iPhone On Feb 20, 2013, at 10:49 AM, "Hiller, Dean" wrote: > I took this jmap dump of cassandra(in production). Before I restarted the > whole production cluster, I had some nodes running compaction and it looked > like all memory had

Re: very confused by jmap dump of cassandra

2013-02-21 Thread aaron morton
Cannot comment too much on the jmap but I can add my general "compaction is hurting" strategy. Try any or all of the following to get to a stable setup, then increase until things go bang. Set concurrent compactors to 2. Reduce compaction throughput by half. Reduce in_memory_compaction_limi

very confused by jmap dump of cassandra

2013-02-20 Thread Hiller, Dean
I took this jmap dump of cassandra(in production). Before I restarted the whole production cluster, I had some nodes running compaction and it looked like all memory had been consumed(kind of like cassandra is not clearing out the caches or memtables fast enough). I am trying to still debug co