Hi,
We are running a production Cassandra version (3.0.14) with 256 tokens
v-node configuration. Occasionally, we see that different nodes show
different ownership for the same key. Only a node restart corrects;
otherwise, it continues to behave in a split-brain.
Say, for example,
*NodeA*
nodetool getendpoints ks1 table1 10
- n1
- n2
- n3
*NodeB*
nodetool getendpoints ks1 table1 10
- n1
- n2
*- n5*
If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The
majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a
few show this issue, and restarting them solves the problem.
To me, it seems I think Cassandra's Gossip cache and StorageService cache
(TokenMetadata) are having some sort of cache coherence.
Anyone has observed this behavior?
Any help would be highly appreciated.
Jaydeep
