Re: Node replacement strategy with AWS EBS

2017-06-14 Thread Hannu Kröger
Hi, So, if it works, great. auto_bootstrap false is not needed when you have system keyspace as also mentioned in the article. Now you are likely to have different tokens then the previous node (unless those were manually configured to match the old node) and repair and cleanup are needed to get

Re: Node replacement strategy with AWS EBS

2017-06-14 Thread Rutvij Bhatt
Thanks again for your help! To summarize for anyone who stumbles onto this in the future, this article covers the procedure well: https://www.eventbrite.com/engineering/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/ It is more or less what Hannu suggested. I carried out the

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread John Hughes
In aws, I just grow the cluster 2x, then shrink away the old nodes via decommission. Mind you I am not dealing with TBs of data, just hundreds of gigs. Also, I have deployment automated with Cloud Formation and Priam. YMMV. On Tue, Jun 13, 2017 at 2:22 PM Cogumelos Maravilha < cogumelosmaravi...@s

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Cogumelos Maravilha
Simplest way of all, if you are using RF>=2 simple terminate the old instance and create a new one. Cheers. On 13-06-2017 18:01, Rutvij Bhatt wrote: > Nevermind, I misunderstood the first link. In this case, the > replacement would just be leaving the listen_address as is (to > InetAddress.getLoc

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Rutvij Bhatt
Nevermind, I misunderstood the first link. In this case, the replacement would just be leaving the listen_address as is (to InetAddress.getLocalHost()) and just start the new instance up as you pointed out in your original answer Hannu. Thanks. On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt wrote

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Rutvij Bhatt
Hannu/Nitan, Thanks for your help so far! From what you said in your first response, I can get away with just attaching the EBS volume to Cassandra and starting it with the old node's private IP as my listen_address because it will take over the token assignment from the old node using the data fi

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Nitan Kainth
Thank you Hannu. > On Jun 13, 2017, at 10:59 AM, Hannu Kröger wrote: > > Hello, > > So the local information about tokens is stored in the system keyspace. Also > the host id and all that. > > Also documented here: > https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addres

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Hannu Kröger
Hello, So the local information about tokens is stored in the system keyspace. Also the host id and all that. Also documented here: https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE If for any reason that causes issues, you can also check this: https://issues.

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Nitan Kainth
Hannu, "Cassandra automatically realizes that have just effectively changed IP address” —> are you sure C* will take care of IP change as is? How will it know which token range to be assigned to this new IP address? > On Jun 13, 2017, at 10:51 AM, Hannu Kröger wrote: > > Cassandra automatica

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Hannu Kröger
Hello, I think that’s not the optimal way to handle it. If you are just attaching the same EBS volume to a new node you can do like this: 1) nodetool drain on old 2) stop cassandra on old 3) Attach EBS to new node 4) Start Cassandra on new node Cassandra automatically realizes that have just eff

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Rutvij Bhatt
Nitan, Yes, that is what I've done. I snapshotted the volume after step 3 and will create a new volume from that snapshot and attach it to the new instance. Curious if I am indeed replacing a node completely, is there any logical difference between snapshot->create->attach vs detach from old->atta

Re: Node replacement strategy with AWS EBS

2017-06-13 Thread Nitan Kainth
Steps are good Rutvij. Step 1 is not mandatory. We snapshot EBS volume and then restored on new node. How are you re-attaching EBS volume without snapshot? I > On Jun 13, 2017, at 10:21 AM, Rutvij Bhatt wrote: > > Hi! > > We're running a Cassandra cluster on AWS. I want to replace an old no

Node replacement strategy with AWS EBS

2017-06-13 Thread Rutvij Bhatt
Hi! We're running a Cassandra cluster on AWS. I want to replace an old node with EBS storage with a new one. The steps I'm following are as follows and I want to get a second opinion on whether this is the right thing to do: 1. Remove old node from gossip. 2. Run nodetool drain 3. Stop cassandra