RE: Problem Replacing a Dead Node

2016-04-21 Thread Jan
used, the data will stream from the decommissioned node. If removetoken is used, the data will stream from the remaining replicas. Hope this helps Jan/ On Thu, 4/21/16, Anubhav Kale wrote: Subject: RE: Problem Replacing a Dead Node To: "

Re: Problem Replacing a Dead Node

2016-04-21 Thread Mir Tanvir Hossain
> > > > *From:* Mir Tanvir Hossain [mailto:mir.tanvir.hoss...@gmail.com] > *Sent:* Thursday, April 21, 2016 11:51 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Problem Replacing a Dead Node > > > > Here is a bit more detail of the whole situation. I am hoping someo

RE: Problem Replacing a Dead Node

2016-04-21 Thread Anubhav Kale
Replacing a Dead Node Here is a bit more detail of the whole situation. I am hoping someone can help me out here. We have a seven node cluster. One the nodes started to have issues but it was running. We decided to add a new node, and remove the problematic node after the new node joins

Re: Problem Replacing a Dead Node

2016-04-21 Thread Mir Tanvir Hossain
Here is a bit more detail of the whole situation. I am hoping someone can help me out here. We have a seven node cluster. One the nodes started to have issues but it was running. We decided to add a new node, and remove the problematic node after the new node joins. However, the new node did not j

Re: Problem Replacing a Dead Node

2016-04-21 Thread Mir Tanvir Hossain
Hi Jeff, thanks for getting back to me. I have gone through the output of nodetool netstats, and it seems all the streams are 100% completed as per the output. What else do you think is going wrong? -Mir On Thu, Apr 21, 2016 at 10:27 AM, Jeff Jirsa wrote: > The keyspace with RF=1 may lose data

Re: Problem Replacing a Dead Node

2016-04-21 Thread Mir Tanvir Hossain
Hi Anubhav, thanks for getting back to me. here is the information that you requested. datastax agent is running on the node. However, in the agent log I see ERROR [clojure-agent-send-off-pool-4] 2016-04-21 17:51:46,055 Can't connect to Cassandra (All host(s) tried for query failed (tried: /10.0.

Re: Problem Replacing a Dead Node

2016-04-21 Thread Jeff Jirsa
The keyspace with RF=1 may lose data, but isn’t blocking the replacement. The most likely cause of the delay is hung streaming. Run `nodetool netstats` on the joining (replacement) node. Do the byte counters change? If not, streaming is hung, and you’ll likely need to restart the process. If so,

RE: Problem Replacing a Dead Node

2016-04-21 Thread Anubhav Kale
Is the datastax-agent running fine on the node ? What does nodetool status and system.log show ? From: Mir Tanvir Hossain [mailto:mir.tanvir.hoss...@gmail.com] Sent: Thursday, April 21, 2016 10:02 AM To: user@cassandra.apache.org Subject: Problem Replacing a Dead Node Hi, I am trying to replace