Re: Too many Cassandra threads waiting!!!

2018-08-02 Thread Joshua Galbraith
Renoy, Thanks. That kernel is new enough to have the patch for the infamous Linux kernel futex bug detailed here: https://groups.google.com/d/topic/mechanical-sympathy/QbmpZxp6C64 To answer your questions above: What you're seeing is likely just normal behavior for Cassandra and is an artifact o

Re: Too many Cassandra threads waiting!!!

2018-08-02 Thread nokia ceph
Hi Joshua, # uname -a Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith < jgalbra...@newrelic.com.invalid> wrote: > Renoy

Re: Too many Cassandra threads waiting!!!

2018-08-02 Thread Joshua Galbraith
Renoy, Out of curiosity, which kernel version are your nodes running? You may find this old message on the mailing list helpful: http://mail-archives.apache.org/mod_mbox/cassandra-user/201602.mbox/%3CCAA=6j0-0vabfan3djfatoxyjwwehpdie67v2wm_u5kaqoro...@mail.gmail.com%3E On Wed, Aug 1, 2018 at 5:3

Re: Too many Cassandra threads waiting!!!

2018-08-01 Thread Elliott Sims
You might have more luck trying to analyze at the Java level, either via a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra tools like "nodetool tpstats" On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph wrote: > Hi, > > i'm having a 5 node cluster with cassandra 3.0.13. > > i

Too many Cassandra threads waiting!!!

2018-08-01 Thread nokia ceph
Hi, i'm having a 5 node cluster with cassandra 3.0.13. i could see the cassandra process has too many threads. *# pstree -p `pgrep java` | wc -l* *453* And almost all of those threads are in *sleeping* state and wait at *# cat /proc/166022/task/1698913/wchan* *futex_wait_queue_me* Some more i