Hi guys, just for the record, in case someone has this issue in the future,
it is a bug, fixed in 1.2.12.
I recommend anyone in this version to upgrade cluster before getting
totally stuck (by adding new machines / DC / altering keyspaces / ...),
well anything using gossip actually. We are going t
Hello Alain,
I'm not sure of the root cause of this item. It may be helpful to use
DEBUG and start the node to see what's happening as well as watch
compaction stats or tpstats to understand what is taxing your system.
The log file you provided shows a large ParNew while replaying commit log
Anyone has any clue of what is happening in our cluster with the given
information?
What other informations could help you to help me :-D ?
2014-06-18 21:07 GMT+02:00 Robert Coli :
> On Wed, Jun 18, 2014 at 5:36 AM, Alain RODRIGUEZ
> wrote:
>
>> We stop the node using : nodetool disablegossi
On Wed, Jun 18, 2014 at 5:36 AM, Alain RODRIGUEZ wrote:
> We stop the node using : nodetool disablegossip && nodetool disablethrift
> && nodetool disablebinary && sleep 10 && nodetool drain && sleep 30 &&
> service cassandra stop
>
The stuff before "nodetool drain" here is redundant and doesn't
This last command was supposed to be a best practice a few years ago, hope
it is still the case. I just added the recent "nodetool disablebinary"
part...
2014-06-18 14:36 GMT+02:00 Alain RODRIGUEZ :
> Thanks a lot for taking time to check the log.
>
> We just switched from 400M to 1600M NEW size
Thanks a lot for taking time to check the log.
We just switched from 400M to 1600M NEW size in the cassandra-env.sh. It
reduced our latency and the PARNEW GC time / second significantly...
(described here
http://tech.shift.com/post/74311817513/cassandra-tuning-the-jvm-for-read-heavy-workloads
)
E
There are several long Parnew pauses that were recorded during startup.
The young gen size looks large too, if I am reading that line correctly.
Did you happen to overwrite the default settings for MAX_HEAP and/or NEW
size in the cassandra-env.sh? The large you gen size, set via the env.sh
file,
Hello
Have you checked the log file to see what's happening during startup
? What caused the rolling restart? Did you preform an upgrade or
change a config?
> On Jun 18, 2014, at 5:40 AM, Alain RODRIGUEZ wrote:
>
> Hi guys
>
> Using 1.2.11, when I try to rolling restart the cluster, any nod
Hi guys
Using 1.2.11, when I try to rolling restart the cluster, any node I restart
makes the whole cluster cpu load to increase, reaching a "red" state in
opscenter (load from 3-4 to 20+). This happens once the node is back online.
The restarted node uses 100 % cpu for 5 - 10 min and sometimes d
