Re: timeouts on counter tables

2017-09-04 Thread Rudi Bruchez
er of cores. 3/ concurrent_counter_writes 4/ Row Cache vs Chunk Cache 5/ change the compaction method to leveled, specifically when using counter columns ?? thanks ! On 3 September 2017 at 20:25, Rudi Bruchez <mailto:r...@babaluga.com>> wrote: Le 30/08/2017 à 05:33, Erick R

Re: timeouts on counter tables

2017-09-04 Thread Rudi Bruchez
r 2017 at 20:25, Rudi Bruchez <mailto:r...@babaluga.com>> wrote: Le 30/08/2017 à 05:33, Erick Ramirez a écrit : Is it possible at all that you may have a data hotspot if it's not hardware-related? It does not seem so, The partition key seems well distributed and

Re: timeouts on counter tables

2017-09-03 Thread Rudi Bruchez
Le 30/08/2017 à 05:33, Erick Ramirez a écrit : Is it possible at all that you may have a data hotspot if it's not hardware-related? It does not seem so, The partition key seems well distributed and the queries update different keys. We have dropped counter_mutation messages in the log : CO

Re: timeouts on counter tables

2017-09-03 Thread Rudi Bruchez
Le 28/08/2017 à 03:30, kurt greaves a écrit : If every node is a replica it sounds like you've got hardware issues. Have you compared iostat to the "normal" nodes? I assume there is nothing different in the logs on this one node? Also sanity check, you are using DCAwareRoundRobinPolicy? ​ Tha

Re: timeouts on counter tables

2017-08-27 Thread Rudi Bruchez
Le 28/08/2017 à 00:11, kurt greaves a écrit : What is your RF? Also, as a side note RAID 1 shouldn't be necessary if you have >1 RF and would give you worse performance 2 + 1 on a backup single node. Consistency one. You're right about RAID 1, if the disk perf is the problem, that might be a

timeouts on counter tables

2017-08-27 Thread Rudi Bruchez
Hello, On a 3 nodes cluster (nodes : 48 procs, 32 Go RAM, SSD), I've timeouts on counter table UPDATEs. One node is specifically slow, generating timeouts. IO bound. iotop shows consistently about 300 Mb/s reads, and writes are around 100 ko/s, changing. The keys seem well distributed. The a