Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-12-30 Thread Robert Coli
On Mon, Dec 29, 2014 at 3:24 PM, mck wrote: > > Especially in CASSANDRA-6285 i see some scary stuff went down. > > But there are no outstanding bugs that we know of, are there? > Right, the question is whether you believe that 6285 has actually been fully resolved. It's relatively plausible tha

Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-12-29 Thread mck
> Perf is better, correctness seems less so. I value latter more than > former. Yeah no doubt. Especially in CASSANDRA-6285 i see some scary stuff went down. But there are no outstanding bugs that we know of, are there? (CASSANDRA-6815 remains just a wrap up of how options are to be presente

Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-12-29 Thread Robert Coli
On Mon, Dec 29, 2014 at 2:03 PM, mck wrote: > We saw an improvement when we switched to HSHA, particularly for our > offline (hadoop/spark) nodes. > Sorry i don't have the data anymore to support that statement, although > i can say that improvement paled in comparison to cross_node_timeout > whi

Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-12-29 Thread mck
> > Should I stick to 2048 or try > > with something closer to 128 or even something else ? 2048 worked fine for us. > > About HSHA, > > I anti-recommend hsha, serious apparently unresolved problems exist with > it. We saw an improvement when we switched to HSHA, particularly for our offline

Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-12-29 Thread Robert Coli
On Mon, Dec 29, 2014 at 2:29 AM, Alain RODRIGUEZ wrote: > Sorry about the gravedigging, but what would be a good start value to tune > "rpc_max_threads" ? > Depends on whether you prefer that clients get a slow thread or none. > I mean, default is unlimited, the value commented is 2048. Native

Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-12-29 Thread Alain RODRIGUEZ
Hi, Sorry about the gravedigging, but what would be a good start value to tune " rpc_max_threads" ? I mean, default is unlimited, the value commented is 2048. Native protocol seems to only allow 128 simultaneous threads. Should I stick to 2048 or try with something closer to 128 or even something

Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-10-29 Thread Peter Haggerty
That definitely appears to be the issue. Thanks for pointing that out! https://issues.apache.org/jira/browse/CASSANDRA-8116 It looks like 2.0.12 will check for the default and throw an exception (thanks Mike Adamson) and also includes a bit more text in the config file but I'm thinking that 2.0.12

Re: 2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-10-29 Thread Duncan Sands
Hi Peter, are you using the hsha RPC server type on this node? If you are, then it looks like rpc_max_threads threads will be allocated on startup in 2.0.11 while this wasn't the case before. This can exhaust your heap if the value of rpc_max_threads is too large (eg if you use the default).

2.0.10 to 2.0.11 upgrade and immediate ParNew and CMS GC storm

2014-10-28 Thread Peter Haggerty
On a 3 node test cluster we recently upgraded one node from 2.0.10 to 2.0.11. This is a cluster that had been happily running 2.0.10 for weeks and that has very little load and very capable hardware. The upgrade was just your typical package upgrade: $ dpkg -s cassandra | egrep '^Ver|^Main' Mainta