Re: Gossip status: hibernate

2016-10-17 Thread Kasper Petersen
I started up a new node and did a "-Dcassandra.replace_address" which I terminated before it could join. This has resulted in one more hibernating node similar to the other one that isn't disappearing after 72 hours. I've added trace level logging to a working node to produce the following (I'm no

Re: Gossip status: hibernate

2016-10-13 Thread Joel Knighton
In the normal case, absolutely. That should happen quickly. I think it is likely the case that you've hit another race condition where this phantom node is not correctly marked as dead. In this case, even if it is removed from some nodes in the cluster, it will get re-added by the node that doesn't

Re: Gossip status: hibernate

2016-10-13 Thread Kasper Petersen
Thanks for the details. I don't know what happened on that node. It's a long time ago I think. I wasn't aware of it earlier. Will a node in hibernating state that failed joining and subsequently was discarded get removed from gossip at some point? On Wed, Oct 12, 2016 at 5:23 PM, Joel Knighton

Re: Gossip status: hibernate

2016-10-12 Thread Joel Knighton
1. A hibernating node is participating in gossip but intentionally hasn't yet joined the ring. The two cases where a node would set a hibernating status are when the node was started with "-Dcassandra.join_ring=False" and has tokens or when the node was started to replace another node (using "-Dcas

Gossip status: hibernate

2016-10-12 Thread Kasper Petersen
Hi, I've recently upgraded our Cassandra cluster from 2.1 to 3.9. By default(?) 3.9 creates a debug.log file containing a ton of lines (a new one every second) with: DEBUG [GossipTasks:1] 2016-10-12 14:43:38,761 Gossiper.java:337 - > Convicting /172.31.137.65 with status hibernate - alive false