Re: Lot of GC on two nodes out of 7

2016-03-06 Thread Anishek Agarwal
ead of decreasing it should help drop (faster) >> rather than promoting to sv/oldgen (slower) ? >> >> >> >> From: Anishek Agarwal >> Reply-To: "user@cassandra.apache.org" >> Date: Thursday, March 3, 2016 at 8:55 PM >> >> To: "user@cassan

Re: Lot of GC on two nodes out of 7

2016-03-04 Thread Jonathan Haddad
m: Anishek Agarwal > Reply-To: "user@cassandra.apache.org" > Date: Thursday, March 3, 2016 at 8:55 PM > > To: "user@cassandra.apache.org" > Subject: Re: Lot of GC on two nodes out of 7 > > Hello, > > Bryan, most of the partition sizes are under 45 KB > >

Re: Lot of GC on two nodes out of 7

2016-03-03 Thread Jeff Jirsa
at 8:55 PM To: "user@cassandra.apache.org" Subject: Re: Lot of GC on two nodes out of 7 Hello, Bryan, most of the partition sizes are under 45 KB I have tried with concurrent_compactors : 8 for one of the nodes still no improvement, I have tried max_heap_Size : 8G, no improveme

Re: Lot of GC on two nodes out of 7

2016-03-03 Thread Anishek Agarwal
o please check whether you have mutations drop in other nodes or not. >> >> >> >> Hope this helps in your cluster too. >> >> >> >> Regards >> >> Amit Singh >> >> *From:* Jonathan Haddad [mailto:j...@jonhaddad.com] >> *S

Re: Lot of GC on two nodes out of 7

2016-03-03 Thread Bryan Cheng
; > Hope this helps in your cluster too. > > > > Regards > > Amit Singh > > *From:* Jonathan Haddad [mailto:j...@jonhaddad.com] > *Sent:* Wednesday, March 02, 2016 9:33 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Lot of GC on two nodes out of 7 > > &g

RE: Lot of GC on two nodes out of 7

2016-03-02 Thread Amit Singh F
To: user@cassandra.apache.org Subject: Re: Lot of GC on two nodes out of 7 Can you post a gist of the output of jstat -gccause (60 seconds worth)? I think it's cool you're willing to experiment with alternative JVM settings but I've never seen anyone use max tenuring threshold of

RE: Lot of GC on two nodes out of 7

2016-03-02 Thread Amit Singh F
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Lot of GC on two nodes out of 7 Can you post a gist of the output of jstat -gccause (60 seconds worth)? I think it's cool you're willing to experiment with alternative JVM settings but I've neve

Re: Lot of GC on two nodes out of 7

2016-03-02 Thread Jonathan Haddad
Can you post a gist of the output of jstat -gccause (60 seconds worth)? I think it's cool you're willing to experiment with alternative JVM settings but I've never seen anyone use max tenuring threshold of 50 either and I can't imagine it's helpful. Keep in mind if your objects are actually reach

Re: Lot of GC on two nodes out of 7

2016-03-02 Thread Alain RODRIGUEZ
It looks like you are doing a good work with this cluster and know a lot about JVM, that's good :-). our machine configurations are : 2 X 800 GB SSD , 48 cores, 64 GB RAM That's good hardware too. With 64 GB of ram I would probably directly give a try to `MAX_HEAP_SIZE=8G` on one of the 2 bad n

Re: Lot of GC on two nodes out of 7

2016-03-02 Thread Anishek Agarwal
Thanks a lot Alian for the details. > `HEAP_NEWSIZE=4G.` is probably far too high (try 1200M <-> 2G) > `MAX_HEAP_SIZE=6G` might be too low, how much memory is available (You > might want to keep this as it or even reduce it if you have less than 16 GB > of native memory. Go with 8 GB if you have a

Re: Lot of GC on two nodes out of 7

2016-03-02 Thread Alain RODRIGUEZ
Hi Anishek, Even if it highly depends on your workload, here are my thoughts: `HEAP_NEWSIZE=4G.` is probably far too high (try 1200M <-> 2G) `MAX_HEAP_SIZE=6G` might be too low, how much memory is available (You might want to keep this as it or even reduce it if you have less than 16 GB of native

Re: Lot of GC on two nodes out of 7

2016-03-02 Thread Anishek Agarwal
also MAX_HEAP_SIZE=6G and HEAP_NEWSIZE=4G. On Wed, Mar 2, 2016 at 1:40 PM, Anishek Agarwal wrote: > Hey Jeff, > > one of the nodes with high GC has 1400 SST tables, all other nodes have > about 500-900 SST tables. the other node with high GC has 636 SST tables. > > the average row size for compa

Re: Lot of GC on two nodes out of 7

2016-03-02 Thread Anishek Agarwal
Hey Jeff, one of the nodes with high GC has 1400 SST tables, all other nodes have about 500-900 SST tables. the other node with high GC has 636 SST tables. the average row size for compacted partitions is about 1640 bytes on all nodes. We have replication factor 3 but the problem is only on two n

Re: Lot of GC on two nodes out of 7

2016-03-01 Thread Jeff Jirsa
Compaction falling behind will likely cause additional work on reads (more sstables to merge), but I’d be surprised if it manifested in super long GC. When you say twice as many sstables, how many is that?. In cfstats, does anything stand out? Is max row size on those nodes larger than on othe