Re: Single slow node dramatically reduces cluster write throughput regardless of CL

2022-12-15 Thread Sarisky, Dan
CASSANDRA-18120 created. On 12/14/2022 3:13 PM, Jeremiah Jordan wrote: I have seen this same behavior in the past as well and came to the same conclusions of where the issue is. It would be good to write this up in a ticket. Giving people the option of using the DynamicEndpointSnitch to orde

Re: Single slow node dramatically reduces cluster write throughput regardless of CL

2022-12-14 Thread Jeremiah Jordan
I have seen this same behavior in the past as well and came to the same conclusions of where the issue is. It would be good to write this up in a ticket. Giving people the option of using the DynamicEndpointSnitch to order batch log replica selection could mitigate this exact issue, but may ha

Single slow node dramatically reduces cluster write throughput regardless of CL

2022-12-14 Thread Sarisky, Dan
We issue writes to Cassandra as logged batches(RF=3, Consistency levels=TWO, QUORUM, or LOCAL_QUORUM) On clusters of any size - a single extremely slow node causes a ~90% loss of cluster-wide throughput using batched writes. We can replicate this in the lab via CPU or disk throttling. I obse