Re: Low compactionthroughput blocks reads?

2016-03-03 Thread Dominik Keil
Hi, seeing this here as well. Basically at some point during a repair within the cluster (incremental, parallel, partitioner range, only one node at a time) some node (sometimes the repairing node, sometimes another) starts piling up READs, i.e. pending reads go through the roof. Despite only one

Re: Low compactionthroughput blocks reads?

2016-03-03 Thread horschi
Hi, we just had such an incident again, and this time we were able to grab a jstack-threaddump. One thread from this stackdump actually shows a ReadVerbHandler calling the RateLimiter. From my understanding this should not be: Thread 30346: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=

Re: Low compactionthroughput blocks reads?

2016-03-03 Thread horschi
Hi, I just found another one. Its basically the same, but I'll post it anyway: Thread 84311: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) - java.lang.Thread.sleep(long, int) @bci=57, line=340 (Compiled frame) - java.util.concurrent.Time

Re: Low compactionthroughput blocks reads?

2016-03-03 Thread Benedict Elliott Smith
Yep, definitely a bug. Introduced by CASSANDRA-9240 (me; mea culpa). I've filed a JIRA for you: CASSANDRA-11301 On 3 March 2016 at 14:10, horschi wrote: > Hi, > > I just found another one. Its basically the same, but I'll post it anyway: > > Thread 84311: (state = BLOCKED) > - java.lang.Thread

Re: Low compactionthroughput blocks reads?

2016-03-03 Thread horschi
Thanks. That's good to know. On Thu, Mar 3, 2016 at 8:48 PM, Benedict Elliott Smith wrote: > Yep, definitely a bug. Introduced by CASSANDRA-9240 (me; mea culpa). > > I've filed a JIRA for you: CASSANDRA-11301 > > On 3 March 2016 at 14:10, horschi wrote: > > > Hi, > > > > I just found another on