Re: Replacing node without shutting down the old node

2023-05-16 Thread Jaydeep Chovatia
Hi Jeff, Do you think this is a good workaround to have in the Cassandra itself until we have CEP-21 available and cleanup as part of the compaction in the Cassandra itself? It can work as follows in C

Re: Replacing node without shutting down the old node

2023-05-16 Thread Runtian Liu
cool, thank you. This looks like a very good setup for us and cleanup should be very fast for this case. On Tue, May 16, 2023 at 5:53 AM Jeff Jirsa wrote: > > In-line > > On May 15, 2023, at 5:26 PM, Runtian Liu wrote: > >  > Hi Jeff, > > I tried the setup with vnode 16 and NetworkTopologyStra

Re: Replacing node without shutting down the old node

2023-05-16 Thread Jeff Jirsa
In-line On May 15, 2023, at 5:26 PM, Runtian Liu wrote:Hi Jeff,I tried the setup with vnode 16 and NetworkTopologyStrategy replication strategy with replication factor 3 with 3 racks in one cluster. When using the new node token as the old node token - 1I had said +1 but you’re right that it’s ac

Re: Replacing node without shutting down the old node

2023-05-15 Thread Runtian Liu
Hi Jeff, I tried the setup with vnode 16 and NetworkTopologyStrategy replication strategy with replication factor 3 with 3 racks in one cluster. When using the new node token as the old node token - 1, I see the new node is streaming from the old node only. And the decom phase of the old node is e

Re: Replacing node without shutting down the old node

2023-05-08 Thread Runtian Liu
I thought the joining node would not participate in quorum? How are we counting things like how many replicas ACK a write when we are adding a new node for expansion? The token ownership won't change until the new node is fully joined right? On Mon, May 8, 2023 at 8:58 PM Jeff Jirsa wrote: > You

Re: Replacing node without shutting down the old node

2023-05-08 Thread Jeff Jirsa
You can't have two nodes with the same token (in the current metadata implementation) - it causes problems counting things like how many replicas ACK a write, and what happens if the one you're replacing ACKs a write but the joining host doesn't? It's harder than it seems to maintain consistency gu

Replacing node without shutting down the old node

2023-05-08 Thread Runtian Liu
Hi all, Sometimes we want to replace a node for various reasons, we can replace a node by shutting down the old node and letting the new node stream data from other replicas, but this approach may have availability issues or data consistency issues if one more node in the same cluster went down. W