Re: bad behavior of my Cassandra cluster

2010-08-04 Thread Jonathan Ellis
Put your commitlog on a different device than the data files. On Tue, Aug 3, 2010 at 10:46 PM, Mingfan Lu wrote: > Hi, >  I have a 4-node cassandra cluster. And I find when the 4 nodes are > flushing memtable and gc at the very similar moment, the throughput > will drop and latency will increase

Re: bad behavior of my Cassandra cluster

2010-08-03 Thread Mingfan Lu
I think you can download the log files now. I am using mmap, at the same time, I use hugepages, so that the jvm heap won't be swapped out, I also swapoff my swap partition According to the https://issues.apache.org/jira/browse/CASSANDRA-1214. I enlarge the memtable threshold I want to reduce the fr

Re: bad behavior of my Cassandra cluster

2010-08-03 Thread Aaron Morton
Just guessing here, but your mem table settings are very high compared to the defaults (64MB and 0.3 Million). It may be that flushing 1GB of mem table data is taking a while. You may also want to look at the  FlushDataBufferSizeInMB and FlushIndexBufferSizeInMB settings in storage-conf.xml.Is the

bad behavior of my Cassandra cluster

2010-08-03 Thread Mingfan Lu
Hi, I have a 4-node cassandra cluster. And I find when the 4 nodes are flushing memtable and gc at the very similar moment, the throughput will drop and latency will increase rapidly and the nodes are dead and up frequently You could download the IOPS variance of data disk (sda here) and sy