Re: nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Mitch Gitman
Sergio and Abe, thanks so much for responding to me so quickly! I managed to figure out the problem and the solution. In the Terraform scripts we used to stand up the EC2 instances, we have a template file for the jmxremote.access file with the content: monitorRole readonly controlRole readwr

Re: nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Sergio
Hey! I would try to spin up a new node and see if the problem occurs on it. If it happens, I would check the history of changes on the cookbook recipe, if you don't find any problem on the new node you might replace all the nodes having problems one by one with a new one and decommission the affect

Re: nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Abe Ratnofsky
Hey Mitch, The security upgrade schedule that your colleague is working on may well be relevant. Is your entire cluster on 3.11.6 or are the failing hosts possibly on a newer version? Abe > On Feb 26, 2023, at 10:38, Mitch Gitman wrote: > >  > We're running Cassandra 3.11.6 on AWS EC2 insta

nodetool suddenly failing with "Access denied!"

2023-02-26 Thread Mitch Gitman
We're running Cassandra 3.11.6 on AWS EC2 instances. These clusters have been running for a few years. We're suddenly noticing now that on one of our clusters the nodetool command is failing on certain nodes but not on others. The failure: nodetool: Failed to connect to '...:7199' - SecurityEx