Re: Cassandra 1.1.6 crash without any exception or error in log

On Thu, Jan 2, 2014 at 10:03 PM, Nitin Sharma wrote: > I would recommend always running cassandra with > -XX:+HeapDumpOnOutofMemoryError. This dumps out a *.hprof file if the > process dies due to OOM > If you do this, be sure to configure the heap dump directory, or ensure that the running dir

Re: Cassandra 1.1.6 crash without any exception or error in log

In this case the Java/Cassandra process never ran out of memory. Rather it had 20% heap free. It is the OS that ran out of memory. This is the side effect of running with large heap. I was aware of the Java's inefficiency wrt large heap but had to keep it due to large bloomfilter. Note we are still

Re: Cassandra 1.1.6 crash without any exception or error in log

I would recommend always running cassandra with -XX:+HeapDumpOnOutofMemoryError. This dumps out a *.hprof file if the process dies due to OOM You can later analyze the hprof files using Eclipse Memory Analyzer (Eclipse MAT ) to figure out root causes and potential lea

Re: Cassandra 1.1.6 crash without any exception or error in log

The root cause turned out to be high heap. The Linux OOM Killer ( http://linux-mm.org/OOM_Killer) killed the process. It took some time to figure out but very interesting. We knew high heap is a problem but had no clue when the actual heap usage was well within limit and the process disappeared. sy

Re: Cassandra 1.1.6 crash without any exception or error in log

On Thu, Jan 2, 2014 at 8:13 AM, Narendra Sharma wrote: > 8 node cluster running in aws. Any pointers where I should start looking? > No kill -9 in history. > You should start looking at instructions as to how to upgrade to at least the top of the 1.1 line... :D =Rob

Cassandra 1.1.6 crash without any exception or error in log

8 node cluster running in aws. Any pointers where I should start looking? No kill -9 in history.