If they went up by 1/7th, could potentially assume it was something related to the snitch not choosing the restarted host. They went up by a lot (2-3x?). What consistency level do you use for reads and writes, and do you have graphs for local reads / hint delivery? (I’m GUESSING that you’re seeing extra read repair or some other multiplier kick in, but it doesn’t make a lot of sense to be honest).
> On Sep 6, 2024, at 9:47 AM, Pradeep Badiger via user > <user@cassandra.apache.org> wrote: > > Hi, > > We are using Cassandra 3.11 with a cluster of 7 nodes and replication of 6 > with most of the default configurations. During a recent maintenance window, > one of the nodes was restarted. The node came back up normal, with no errors > of any sort. But when the application started using the cluster, we found > below-normal disk io read rates on the node that was restarted, and other > nodes in the cluster reported above-normal disk io read rates. This > difference became significant causing alerts to get reported by the > monitoring system. As a measure to resolve the issue the application was > stopped and the entire cluster was restarted after which all 7 nodes reported > almost the same read rates. > > <image001.png> > Figure 1 - After the node 53 was restarted. > > > <image002.png> > Figure 2 - After the entire cluster restart. > > The node in question was not down for a very long time. Is there any specific > reason the read rates would differ like this? Is there a way to resolve this > without restarting the entire cluster? > > Thanks, > Pradeep V.B. > This email and any files transmitted with it are confidential, proprietary > and intended solely for the individual or entity to whom they are addressed. > If you have received this email in error please delete it immediately.
