Re: Kernel keeps killing cassandra process - OOM

2014-03-27 Thread prem yadav
Hi all, thanks for the help. the issue happened again. I looked into Romain's suggestion and yes, it was VMware balloning. So, i will not be updating the jira ticket as I am not sure if the fix actually helped or not. Thanks, Prem On Wed, Mar 26, 2014 at 6:54 PM, Robert Coli wrote: > On Wed, M

Re: Kernel keeps killing cassandra process - OOM

2014-03-26 Thread Robert Coli
On Wed, Mar 26, 2014 at 8:35 AM, prem yadav wrote: > Thanks Robert. That seems to be the issue. however the fix mentioned there > doesn't work. I downgraded Java to jdk6_37 and that seems to have done the > trick. Thanks for pointing me to that Jira ticket. > If the workaround on that ticket doe

Re: Kernel keeps killing cassandra process - OOM

2014-03-26 Thread prem yadav
Thanks Robert. That seems to be the issue. however the fix mentioned there doesn't work. I downgraded Java to jdk6_37 and that seems to have done the trick. Thanks for pointing me to that Jira ticket. On Mon, Mar 24, 2014 at 6:48 PM, Robert Coli wrote: > On Mon, Mar 24, 2014 at 4:11 AM, prem ya

Re: Kernel keeps killing cassandra process - OOM

2014-03-24 Thread Robert Coli
On Mon, Mar 24, 2014 at 4:11 AM, prem yadav wrote: > the nodes die *without * being under any load. Completely idle. > https://issues.apache.org/jira/browse/CASSANDRA-6541 ? =Rob

Re: Kernel keeps killing cassandra process - OOM

2014-03-24 Thread Romain HARDOUIN
e : prem yadav > A : user@cassandra.apache.org, > Date : 24/03/2014 12:12 > Objet : Re: Kernel keeps killing cassandra process - OOM > > the nodes die without being under any load. Completely idle. > And 4 GB system memory is not low. or is it? > I have tried tweaking the overc

Re: Kernel keeps killing cassandra process - OOM

2014-03-24 Thread prem yadav
> 22:25:30 : > > > De : "Laing, Michael" > > A : user@cassandra.apache.org, > > Date : 22/03/2014 22:26 > > Objet : Re: Kernel keeps killing cassandra process - OOM > > > > You might want to look at: > > > > http://www.opensourceconnections.com/2013/08/31/building-the- > > perfect-cassandra-test-environment/ >

Re: Kernel keeps killing cassandra process - OOM

2014-03-24 Thread Romain HARDOUIN
en if they are not loaded? If the nodes are VM you have to pay attention to hypervisor memory overcommit. "Laing, Michael" a écrit sur 22/03/2014 22:25:30 : > De : "Laing, Michael" > A : user@cassandra.apache.org, > Date : 22/03/2014 22:26 > Objet : Re: Kernel

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread Laing, Michael
You might want to look at: http://www.opensourceconnections.com/2013/08/31/building-the-perfect-cassandra-test-environment/ On Sat, Mar 22, 2014 at 4:38 PM, Brian Flad wrote: > Is there anything else running on the boxes? Can you show us the output of > ps aux for the Cassandra process so we c

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread prem yadav
Also we use datastax. The version cassandra-1.1.9 doesn't work with java 7 On Sat, Mar 22, 2014 at 9:09 PM, prem yadav wrote: > The output of ps waux . Also there is no load on cluster. None > > USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND > root 1 0.0 0.0

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread prem yadav
The output of ps waux . Also there is no load on cluster. None USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 19224 1076 ?Ss Mar19 0:01 /sbin/init root 2 0.0 0.0 0 0 ?SMar19 0:00 [kthreadd] root

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread Brian Flad
Is there anything else running on the boxes? Can you show us the output of ps aux for the Cassandra process so we can see the -xmx, etc? JDK 7 may help, even if you cannot upgrade Cassandra yet (which I would really recommend since it moves items off-heap). On Sat, Mar 22, 2014 at 4:01 PM, prem y

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread prem yadav
Upgrading is not possible right now. Any other suggestions guys? I have already tried reducing the number of rpc threads. Also tried reducing the linux kernel overcommit. On Sat, Mar 22, 2014 at 5:44 PM, Laing, Michael wrote: > I ran into the same problem some time ago. > > Upgrading to Cassandr

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread Laing, Michael
I ran into the same problem some time ago. Upgrading to Cassandra 2, jdk 1.7, and default parameters fixed it. I think the jdk change was the key for my similarly small memory cluster. ml On Sat, Mar 22, 2014 at 1:36 PM, prem yadav wrote: > Michael, no memory constraints. System memory is 4

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread prem yadav
Michael, no memory constraints. System memory is 4 GB and Cassandra run on default. On Sat, Mar 22, 2014 at 5:32 PM, prem yadav wrote: > Its Oracle jdk 1.6. > Robert, any fix that you know of which went into 1.2.15 for this > particular issue? > > > On Sat, Mar 22, 2014 at 4:50 PM, Robert Coli

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread prem yadav
Its Oracle jdk 1.6. Robert, any fix that you know of which went into 1.2.15 for this particular issue? On Sat, Mar 22, 2014 at 4:50 PM, Robert Coli wrote: > On Sat, Mar 22, 2014 at 7:48 AM, prem yadav wrote: > >> But, the cassandra process keeps getting killed due to OOM. Cassandra >> version

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread Robert Coli
On Sat, Mar 22, 2014 at 7:48 AM, prem yadav wrote: > But, the cassandra process keeps getting killed due to OOM. Cassandra > version in use is 1.1.9. > Try using 1.2.15, instead? =Rob

Re: Kernel keeps killing cassandra process - OOM

2014-03-22 Thread Michael Dykman
What JVM are you running on? What, if any, memory constraints are you passing to the process? On Mar 22, 2014 10:48 AM, "prem yadav" wrote: > Hi, > I have a 3 node cassandra test cluster. The nodes have 4 GB total memory/2 > cores. Cassndra run with all default settings. > But, the cassandra pro

Kernel keeps killing cassandra process - OOM

2014-03-22 Thread prem yadav
Hi, I have a 3 node cassandra test cluster. The nodes have 4 GB total memory/2 cores. Cassndra run with all default settings. But, the cassandra process keeps getting killed due to OOM. Cassandra version in use is 1.1.9. here are the settings in use: compaction_throughput_mb_per_sec: 16 row_cache_