Re: RC1 - joining cluster

2021-05-10 Thread Kane Wilson
Well, that sounds like a dangerous sequence of events, but should have worked in the end regardless. Probably next time give it a bit more time and keep an eye on netstats and compactionstats. raft.so - Cassandra consulting, support, and managed services On Mon, May 10, 2021 at 10:23 PM Joe Obe

Re: Counter errors - RC1

2021-05-10 Thread Kane Wilson
Seems like some of your nodes are overloaded. Is it intentional that some of your nodes have varying numbers of tokens? It seems like some of your nodes are overloaded, potentially at least #RF of them. If nodes are heavily overloaded GC tuning generally won't help much, you're best off starting b

Counter errors - RC1

2021-05-10 Thread Joe Obernberger
Hi all - I'm getting the following error on RC1: WARN  [Messaging-EventLoop-3-23] 2021-05-10 17:29:12,431 NoSpamLogger.java:95 - /172.16.100.39:7000->/172.16.100.248:7000-URGENT_MESSAGES-e8d21588 dropping message of type FAILURE_RSP whose timeout expired before reaching the network ERROR [Co

Re: RC1 - joining cluster

2021-05-10 Thread Joe Obernberger
Hi - I waited 3 hours.  It was syncing up data; I could see network traffic, but then it stopped.  I didn't check netstats, but I did check compactionstats and there were no pending tasks. I then set auto_bootstrap to false on both new machines and they joined.  Then ran a repair. -Joe O