Cross-Node Latency Issues

2024-10-24 Thread Naman kaushik
Hello everyone, We are currently using Cassandra 4.1.3 in a two-data-center cluster. Recently, we observed cross-node latency spikes of 3-4 seconds in one of our data centers. Below are the relevant logs from all three nodes in this DC: DEBUG [ScheduledTasks:1] 2024-10-20 02:46:43,164 MonitoringT

Re: Cross-Node Latency Issues

2024-10-24 Thread Bowen Song via user
Even big cloud providers, like GCP and AWS, can have temporary and minor network issues every now and then. If it was the result of an increase in packet loss for a short duration, TCP retransmission may be of interest. Have a look at /proc/net/netstat on Linux and you will find the relevant me

Re: Cross-Node Latency Issues

2024-10-24 Thread Stéphane Alleaume
Can you share DDL about table and the keyspace related ? Any interesting informations about the queries and CL associated ? Are you running read repair between DC ? NTP (time server) is well configured for all nodes in the 2 datacenters ? No antivirus activity on these nodes ? Kind regards S

Re: [External]Cross-Node Latency Issues

2024-10-24 Thread Naman kaushik
Hi jiri, Thank you for your feedback. I understand that using ALLOW FILTERING can lead to longer processing times and impact our SLA. However, I want to clarify that the queries in question are running only on the partition key. The WHERE clause is utilizing only the partition column for filterin

Re: Cross-Node Latency Issues

2024-10-24 Thread Bowen Song via user
Can you be more explicit about the "latency metrics from Grafana" you looked at? What percentile latencies were you looking at? Any aggregation used? You can post the underlying queries used for the dashboard if that's easier than explaining it. In general you should only care about the max, no

unsubscribe

2024-10-24 Thread Geukens, Wim (ext)
Worldline, Cardlink, GoPay and Santeos are registered trademarks and trade names owned by the Worldline Group. This e-mail and any documents attached are confidential and intended solely for the addressee. It may also be privileged. If you are not the intended recipient of this e-mail, you are

Re: Tombstone Generation in Cassandra 4.1.3 Despite No Update/Delete Operations

2024-10-24 Thread Bowen Song via user
Is one tombstone scanned per query causing any issue? I mean real issues, not the scanning of tombstone itself. On 24/10/2024 04:56, Naman kaushik wrote: Thanks everyone for your responses. We have columns with |list| and |list| types, and after using |sstabledump|, we found that the tombston

Re: Cross-Node Latency Issues

2024-10-24 Thread manish khandelwal
HI Naman If you are querying on non partition key (which seems to be the case here), then please know that it is an anti pattern for Cassandra. For small dataset it may work but for large dataset it may take longer time or generally timeout. Reason for this is Cassandra scans each record if querie