Re: Cross-Node Latency Issues

2024-10-25 Thread Naman kaushik
Hi Manish, Just to clarify, our queries are using only the partition key column, with no additional filtering. It appears Cassandra is automatically logging ALLOW FILTERING, which may be misleading. Since we’re operating strictly on the partition key, we shouldn’t face the usual performance issues

Re: Cross-Node Latency Issues

2024-10-25 Thread MyWorld
Hi, This was a one time issue for which we are looking for the RCA. Generally P99 latencies of all the tables are less than 12ms. There was a few ms jump in P99 on one of the node at this time at coordinator level. The CL is Local_Quorum. Another error we noticed in system log at the same time on

Cross-Node Latency Issues

2024-10-24 Thread Naman kaushik
Hello everyone, We are currently using Cassandra 4.1.3 in a two-data-center cluster. Recently, we observed cross-node latency spikes of 3-4 seconds in one of our data centers. Below are the relevant logs from all three nodes in this DC: DEBUG [ScheduledTasks:1] 2024-10-20 02:46:43,164 MonitoringT

Re: Cross-Node Latency Issues

2024-10-24 Thread Bowen Song via user
Even big cloud providers, like GCP and AWS, can have temporary and minor network issues every now and then. If it was the result of an increase in packet loss for a short duration, TCP retransmission may be of interest. Have a look at /proc/net/netstat on Linux and you will find the relevant me

Re: Cross-Node Latency Issues

2024-10-24 Thread Stéphane Alleaume
Can you share DDL about table and the keyspace related ? Any interesting informations about the queries and CL associated ? Are you running read repair between DC ? NTP (time server) is well configured for all nodes in the 2 datacenters ? No antivirus activity on these nodes ? Kind regards S

Re: [External]Cross-Node Latency Issues

2024-10-24 Thread Naman kaushik
of the EmbedIT. Do not distribute outside of the organization. > -- > *Od:* Naman kaushik > *Odesláno:* čtvrtek, října 24, 2024 6:26:27 dop. > *Komu:* user@cassandra.apache.org > *Předmět:* [External]Cross-Node Latency Issues > > This message is from an EXTERNAL S

Re: Cross-Node Latency Issues

2024-10-24 Thread Bowen Song via user
Can you be more explicit about the "latency metrics from Grafana" you looked at? What percentile latencies were you looking at? Any aggregation used? You can post the underlying queries used for the dashboard if that's easier than explaining it. In general you should only care about the max, no

Re: Cross-Node Latency Issues

2024-10-24 Thread manish khandelwal
HI Naman If you are querying on non partition key (which seems to be the case here), then please know that it is an anti pattern for Cassandra. For small dataset it may work but for large dataset it may take longer time or generally timeout. Reason for this is Cassandra scans each record if querie

Re: Cross-Node Latency Issues

2024-10-23 Thread MyWorld
How many sstables per read ? -- >> 1 to 4 sstables Are your partitions unbounded ? >> No What max size are the partitions ? >> P99 varies from few bytes to 70KB while max partition of tables varies from few bytes to 900 KB On Thu, Oct 24, 2024 at 11:00 AM Stéphane Alleaume wrote: > Hi, > > How

Re: Cross-Node Latency Issues

2024-10-23 Thread Stéphane Alleaume
Hi, How many sstables per read ? Are your partitions unbounded ? What max size are the partitions ? Kind regards Stéphane Le 24 octobre 2024 06:25:05 GMT+02:00, Naman kaushik a écrit : >Hello everyone, > >We are currently using Cassandra 4.1.3 in a two-data-center cluster. >Recently, we

Re: [External]Cross-Node Latency Issues

2024-10-23 Thread Jiri Steuer (EIT)
item's classification is Internal. It was created by and is in property of the EmbedIT. Do not distribute outside of the organization. Od: Naman kaushik Odesláno: čtvrtek, října 24, 2024 6:26:27 dop. Komu: user@cassandra.apache.org Předmět: [External]Cross-N