Re: Bootstrapping fails with < 128GB RAM ...

2018-02-16 Thread Jürgen Albersdorfer
Hi Jon, I was able to get a heapdump. - I created a JIRA and attached as much as details as possible. https://issues.apache.org/jira/browse/CASSANDRA-14239 The heapdump is 42GB in Size. I will keep it - if you need more information please don't hesitate to let me know. thanks, Jürgen 2018-02-09

Re: Bootstrapping fails with < 128GB RAM ...

2018-02-09 Thread Jürgen Albersdorfer
Hi Jon, should I register to the JIRA and open an Issue or will you do so? I'm currently trying to bootstrap another node - with 100GB RAM, this time, and I'm recording Java Heap Memory over time via Jconsole, Top Threads and do monitoring the debug.log. There, in the debug.log, I can see, that th

Re: Bootstrapping fails with < 128GB RAM ...

2018-02-07 Thread Jon Haddad
It would be extremely helpful to get some info about your heap. At a bare minimum, a histogram of the heap dump would be useful, but ideally a full heap dump would be best. jmap -dump:live,format=b,file=heap.bin PID Taking a look at that in YourKit should give some pretty quick insight into

Re: Bootstrapping fails with < 128GB RAM ...

2018-02-07 Thread Nicolas Guyomar
Ok then, following up on the wild guess : because you have quite a lot of concurrent compactors, maybe it is too much concurrent compactions for the jvm to deal with (taking into account that your load average of 106 seems really high IMHO) 55Gb of data is not that much, you can try to reduce thos

Re: Bootstrapping fails with < 128GB RAM ...

2018-02-07 Thread Jürgen Albersdorfer
Hi Nicolas, Do you know how many sstables is this new node suppose to receive ? If I can find out this via nodetool netstats, then this would be 619 as following: # nodetool netstats Bootstrap b95371e0-0c0a-11e8-932b-f775227bf21c /192.168.1.215 - Receiving 71 files, 7744612158 bytes total.

Re: Bootstrapping fails with < 128GB RAM ...

2018-02-07 Thread Nicolas Guyomar
Hi Jurgen, It does feel like some OOM during boostrap from previous C* v2, but that sould be fixed in your version. Do you know how many sstables is this new node suppose to receive ? Juste a wild guess : it may have something to do with compaction not keeping up because every other nodes are st

Bootstrapping fails with < 128GB RAM ...

2018-02-07 Thread Jürgen Albersdorfer
Hi, I always face an issue when bootstrapping a Node having less than 184GB RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster. During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap Old Gen which gets not significantly freed any more. I know that JVM collects on Old Gen