Hello community, I came across the following post: https://lists.apache.org/thread/t94jjc4j8dfd9sg5of9283mqcy3y3bgq This gives me a good explanation of the log. I will try to check the storage of the problematic datacenters. I will coma back if I have any good findings.
BR MK From: Michail Kotsiouros via user <user@cassandra.apache.org> Sent: Tuesday, September 27, 2022 17:36 To: user@cassandra.apache.org Subject: Log messages "MUTATION messages were dropped" in system.log Hello Cassandra community, I see the following log lines in the Cassandra nodes of 2 datacenters that participate in a cluster of 6 datacenters. INFO [ScheduledTasks:1] 2022-09-12 17:19:44,422 MessagingService.java:1236 - MUTATION messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 9140 ms<https://protect2.fireeye.com/v1/url?k=31323334-501cfaf3-313273af-454445554331-29833d79f9680915&q=1&e=25113adf-a158-4321-a644-3c79d2792963&u=http%3A%2F%2Fmessagingservice.java%3A1236%2F> >From the information I have found online, I see that those logs indicate a >resource issue on the Cassandra nodes. I am trying to troubleshoot what could >be the bottleneck. Can you, please, help me interpret the log line. So far I know that MUTATION means that it is write operations for already existing keys. What internal means? Does it indicate that the write operation is originated from a coordinator node residing in the same datacenter? What cross node means? Does it indicate that the write operation is originated from a coordinator node residing in another datacenter? What dropped latency means? Is it the time that the message spent in some queue before dropped? I would like to know that so as to understand the impact observed on client application. Thank you in advance!!! I find really useful all the information found in different posts!!! BR Michail
