Re: Cassandra timeout whereas it is not much busy

2013-01-29 Thread Nicolas Lalevée
Le 29 janv. 2013 à 08:08, aaron morton a écrit : >> From what I could read there seems to be a contention issue around the >> flushing (the "switchlock" ?). Cassandra would then be slow, but not using >> the entire cpu. I would be in the strange situation I was where I reported >> my issue in

Re: Cassandra timeout whereas it is not much busy

2013-01-28 Thread aaron morton
> From what I could read there seems to be a contention issue around the > flushing (the "switchlock" ?). Cassandra would then be slow, but not using > the entire cpu. I would be in the strange situation I was where I reported my > issue in this thread. > Does my theory makes sense ? If you are

Re: Cassandra timeout whereas it is not much busy

2013-01-28 Thread Nicolas Lalevée
I did some testing, I have a theory. First, we have it seems "a lot" of CF. And two are particularly every hungry in RAM, consuming a quite big amount of RAM for the bloom filters. Cassandra do not force the flush of the memtables if it has more than 6G of Xmx (luckily for us, this is the maxim

Re: Cassandra timeout whereas it is not much busy

2013-01-23 Thread Nicolas Lalevée
Le 22 janv. 2013 à 21:50, Rob Coli a écrit : > On Wed, Jan 16, 2013 at 1:30 PM, Nicolas Lalevée > wrote: >> Here is the long story. >> After some long useless staring at the monitoring graphs, I gave a try to >> using the openjdk 6b24 rather than openjdk 7u9 > > OpenJDK 6 and 7 are both counter

Re: Cassandra timeout whereas it is not much busy

2013-01-22 Thread Rob Coli
On Wed, Jan 16, 2013 at 1:30 PM, Nicolas Lalevée wrote: > Here is the long story. > After some long useless staring at the monitoring graphs, I gave a try to > using the openjdk 6b24 rather than openjdk 7u9 OpenJDK 6 and 7 are both counter-recommended with regards to Cassandra. I've heard reports

Re: Cassandra timeout whereas it is not much busy

2013-01-22 Thread aaron morton
> I have seen logs about that. I didn't worry much, since the GC of the jvm was > not under pressure. When cassandra logs a ParNew event from the GCInspector that is time the server is paused / frozen. CMS events have a very small pause, but they are taking a non trivial amount of CPU time. I

Re: Cassandra timeout whereas it is not much busy

2013-01-21 Thread Nicolas Lalevée
Le 17 janv. 2013 à 05:00, aaron morton a écrit : > Check the disk utilisation using iostat -x 5 > If you are on a VM / in the cloud check for CPU steal. > Check the logs for messages from the GCInspector, the ParNew events are times > the JVM is paused. I have seen logs about that. I didn't w

Re: Cassandra timeout whereas it is not much busy

2013-01-16 Thread aaron morton
Check the disk utilisation using iostat -x 5 If you are on a VM / in the cloud check for CPU steal. Check the logs for messages from the GCInspector, the ParNew events are times the JVM is paused. Look at the times dropped messages are logged and try to correlate them with other server events.

Cassandra timeout whereas it is not much busy

2013-01-16 Thread Nicolas Lalevée
Hi, I have a strange behavior I am not able to understand. I have 6 nodes with cassandra-1.0.12. Each nodes have 8G of RAM. I have a replication factor of 3. --- my story is maybe too long, trying shorter here, while saving what I wrote in case someone has patience to read my bad e