Re: too many warnings of Heap is full

2013-02-01 Thread Guillermo Barbero
> What is the cardinality like on these indexes? Can you provide the > schema creation for these two column families? This is the schema of the CFs: create column family CF_users with comparator = UTF8Type and column_metadata = [

Re: too many warnings of Heap is full

2013-01-31 Thread Nate McCall
> > There are 4 secondary indexes for users and 3 for messages. I've seen > something odd here: the index of the CFs is also stored in a column > AND is also indexed. There isn't any query by that secondary index. I > should probably erase the index, shouldn't I? What is the cardinality like on th

Re: too many warnings of Heap is full

2013-01-31 Thread Guillermo Barbero
Hi, replies to your answers: The CFs that seems to flush more often (users and messages) do not have multiget queries... There are 4 secondary indexes for users and 3 for messages. I've seen something odd here: the index of the CFs is also stored in a column AND is also indexed. There isn't any q

Re: too many warnings of Heap is full

2013-01-31 Thread Alain RODRIGUEZ
@Bryan "Other space is used but can be more easily controlled by tuning for - memtable - compaction - key cache - *row cache*" Isn't row cache stored off-heap since a while ? 2013/1/31 Bryan Talbot > On Wed, Jan 30, 2013 at 2:44 PM, Guillermo Barbero < > guillermo.barb...@spotbros.com> wrote:

Re: too many warnings of Heap is full

2013-01-30 Thread Bryan Talbot
On Wed, Jan 30, 2013 at 2:44 PM, Guillermo Barbero < guillermo.barb...@spotbros.com> wrote: > WARN [MemoryMeter:1] 2013-01-30 21:37:48,079 Memtable.java (line 202) > setting live ratio to maximum of 64.0 instead of 751.6512549537648 > > This looks interesting. Doesn't this mean that the ratio of

Re: too many warnings of Heap is full

2013-01-30 Thread Nate McCall
Your latencies and distribution look fine. How big/what types of queries are you issuing? Are you issuing a lot of large multigets? Also, do either of these column families have secondary indexes? On Wed, Jan 30, 2013 at 2:59 PM, Guillermo Barbero wrote: > Iep, > > I missed the attachment...

Re: too many warnings of Heap is full

2013-01-30 Thread Nate McCall
What's the output of nodetool cfstats for those 2 column families on cassNode2 and cassNode3? And what is the replication factor for this cluster? Per the previous reply, nodetool ring should show each of your nodes with ~16.7% of the data if well balanced. Also, the auto-detection for memory siz

Re: too many warnings of Heap is full

2013-01-30 Thread Bryan Talbot
My guess is that those one or two nodes with the gc pressure also have more rows in your big CF. More rows could be due to imbalanced distribution if your'e not using a random partitioner or from those nodes not yet removing deleted rows which other nodes may have done. JVM heap space is used for