Re: Out of control memory consumption

2011-02-11 Thread Robert Coli
On Thu, Feb 10, 2011 at 9:17 AM, Huy Le wrote: > Yes, we had setting at 75 but JVM did not have enough time to do GC, so it > abort GC'ing.   We lowered it to 50, but still had issue, so we lowered it > again to 35. If you lower CMSInitiatingOccupancyFraction below the size of your typical workin

Re: Out of control memory consumption

2011-02-10 Thread Jonathan Ellis
Sounds like you need to use a larger heap or put less stuff in it (memtables, caches). On Thu, Feb 10, 2011 at 11:17 AM, Huy Le wrote: > Yes, we had setting at 75 but JVM did not have enough time to do GC, so it > abort GC'ing.   We lowered it to 50, but still had issue, so we lowered it > again

Re: Out of control memory consumption

2011-02-10 Thread Huy Le
Yes, we had setting at 75 but JVM did not have enough time to do GC, so it abort GC'ing. We lowered it to 50, but still had issue, so we lowered it again to 35. On Thu, Feb 10, 2011 at 12:11 PM, Oleg Anastasyev wrote: > Huy Le springpartners.com> writes: > > > Our CMS settings are:-XX:

Re: Out of control memory consumption

2011-02-10 Thread Oleg Anastasyev
Huy Le springpartners.com> writes: > Our CMS settings are:    -XX:CMSInitiatingOccupancyFraction=35 \    -XX:+UseCMSInitiatingOccupancyOnly \  > Occupancy Fraction = 35 seems very low value. You instructed GC to make collection as soon as memory usage is at 35% - i.e. about 1G. This see

Re: Out of control memory consumption

2011-02-10 Thread Huy Le
We use Cassandra version 0.6.11. Our cache size is very small. 11 out of 12 servers have used heap size less than 500MB of 3GB allocated. Just one server that had memory usage run out of control. The issue is isolated. It turn out that one CF has a row with compacted row size of 50MB. And this

Re: Out of control memory consumption

2011-02-09 Thread Robert Coli
On Wed, Feb 9, 2011 at 11:04 AM, Huy Le wrote: > Memory usage grows overtime. It is relatively typical for caches to exert memory pressure over time as they fill. What are your cache settings, for how many columnfamilies, and with what sized memtables? What version of Cassandra? =Rob

Re: Out of control memory consumption

2011-02-09 Thread Huy Le
> > To be clear: You are not talking about the size of the Java process in > top, but the actual amount of heap used as reported by the JVM via > jmx/jconsole/etc? > > This is memory usage shows in JMX that we are talking about. > Is the memory amount of memory that you consider high, the heap

Re: Out of control memory consumption

2011-02-09 Thread Huy Le
> > If the heap usages continues to grow an OOM will eventually be thrown. > Are you experiencing OOMs on these boxes? If you are not OOMing, then > what problem are you experiencing (excessive CPU use garbage collection > for one example)? > > > No OOM. The JVM just too busy doing GC when the us

Re: Out of control memory consumption

2011-02-09 Thread Peter Schuller
(If you're looking at e.g. jconsole graphs a screenshot of the graph would not hurt.) -- / Peter Schuller

Re: Out of control memory consumption

2011-02-09 Thread Peter Schuller
> We are 12-server cluster.  We use random partitioner with manually generated > server tokens.  Memory usage on one server keeps growing out of control.  We > ran flush and cleared key and row caches but and ran GC but heap memory > usage won't go down.  The only way to heap memory usage to go dow

Re: Out of control memory consumption

2011-02-09 Thread Chris Burroughs
On 02/09/2011 11:15 AM, Huy Le wrote: > There is already an email thread on memory issue on this email list, but I > creating a new thread as we are experiencing a different memory consumption > issue. > > We are 12-server cluster. We use random partitioner with manually generated > server tokens

Out of control memory consumption

2011-02-09 Thread Huy Le
Hi, There is already an email thread on memory issue on this email list, but I creating a new thread as we are experiencing a different memory consumption issue. We are 12-server cluster. We use random partitioner with manually generated server tokens. Memory usage on one server keeps growing o