Re: Cassandra process exiting mysteriously

2014-08-26 Thread Or Sher
Hi Clint, I think I kind of found the reason for my problem, I doubt you have the exact same problem but here it is: We're using Zabbix as our monitoring system and it uses /usr/bin/at to schedule it monitoring runs. Every time the "at" command adds another scheduled task, it send a kill signal to

Re: Cassandra process exiting mysteriously

2014-08-12 Thread Or Sher
Will do the same! Thanks, Or. On Tue, Aug 12, 2014 at 6:47 PM, Clint Kelly wrote: > Hi Or, > > For now I removed the test that was failing like this from our suite > and made a note to revisit it in a couple of weeks. Unfortunately I > still don't know what the issue is. I'll post here if I f

Re: Cassandra process exiting mysteriously

2014-08-12 Thread Clint Kelly
Hi Or, For now I removed the test that was failing like this from our suite and made a note to revisit it in a couple of weeks. Unfortunately I still don't know what the issue is. I'll post here if I figure out it (please do the same!). My working hypothesis now is that we had some kind of OOM

Re: Cassandra process exiting mysteriously

2014-08-12 Thread Or Sher
Clint, did you find anything? I just noticed it happens to us too on only one node in our CI cluster. I don't think there is a special usage before it happens... The last line in the log before the shutdown lines in at least an hour before.. We're using C* 2.0.9. On Thu, Aug 7, 2014 at 12:49 AM,

Re: Cassandra process exiting mysteriously

2014-08-06 Thread Clint Kelly
Hi Rob, Thanks for the clarification; this is really useful. I'll run some experiments to see if the problem is a JVM OOM on our build machine. Best regards, Clint On Wed, Aug 6, 2014 at 1:14 PM, Robert Coli wrote: > On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli wrote: >> >> On Wed, Aug 6, 2014

Re: Cassandra process exiting mysteriously

2014-08-06 Thread Robert Coli
On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli wrote: > On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands > wrote: > >> this doesn't look like an OOM to me. If the kernel OOM kills Cassandra >> then Cassandra instantly vaporizes, and there will be nothing in the >> Cassandra logs (you will find informa

Re: Cassandra process exiting mysteriously

2014-08-06 Thread Robert Coli
On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands wrote: > this doesn't look like an OOM to me. If the kernel OOM kills Cassandra > then Cassandra instantly vaporizes, and there will be nothing in the > Cassandra logs (you will find information about the OOM in the system logs > though, eg in dmesg).

Re: Cassandra process exiting mysteriously

2014-08-06 Thread Clint Kelly
Hi Duncan, Thanks for your help. I am at a loss as to what is causing this process to stop then. I would not expect the Cassandra process to finish until my code calls Process#destroy, but it seems to non-deterministically stop much earlier sometimes. FWIW I have seen failures on another machin

Re: Cassandra process exiting mysteriously

2014-08-06 Thread Duncan Sands
Hi Clint, INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,903 ThriftServer.java (line 141) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14:51,920 Server.java (line 182) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-08-05 19:14

Re: Cassandra process exiting mysteriously

2014-08-05 Thread Clint Kelly
HI Kevin, Thanks for your reply. That is what I assumed, but some of the posts I read on Stack Overflow (e.g., the one that I referenced in my mail) suggested otherwise. I was just curious if others had experienced OOM problems that weren't logged or if there were other common culprits. Best re

Re: Cassandra process exiting mysteriously

2014-08-05 Thread Kevin Burton
If there is an oom it will be in the logs. On Aug 5, 2014 8:17 PM, "Clint Kelly" wrote: > Hi everyone, > > For some integration tests, we start up a CassandraDaemon in a > separate process (using the Java 7 ProcessBuilder API). All of my > integration tests run beautifully on my laptop, but one

Cassandra process exiting mysteriously

2014-08-05 Thread Clint Kelly
Hi everyone, For some integration tests, we start up a CassandraDaemon in a separate process (using the Java 7 ProcessBuilder API). All of my integration tests run beautifully on my laptop, but one of them fails on our Jenkins cluster. The failing integration test does around 10k writes to diffe