Re: Issue with removing a node and adding it back

2015-03-30 Thread Robert Coli
On Fri, Mar 27, 2015 at 4:27 PM, Shiwen Cheng wrote: > Thanks Robert! > Yes I tried what you said: clean the data and re-bootstrap. But still it > failed, once at the point of 600GB transferred and once at 1.1TB :( > 1) figure out what is making your streams die (usually either flaky network (A

Re: Issue with removing a node and adding it back

2015-03-27 Thread Shiwen Cheng
Thanks Robert! Yes I tried what you said: clean the data and re-bootstrap. But still it failed, once at the point of 600GB transferred and once at 1.1TB :( But I could see following exceptions from time to time: = java.io.IOException: net.jpountz.lz4.LZ4Exception: Error decodin

Re: Issue with removing a node and adding it back

2015-03-26 Thread Robert Coli
On Thu, Mar 26, 2015 at 11:31 AM, Shiwen Cheng wrote: > I encountered an issue by removing and adding back a node. > You are encountering a failed/hung bootstrap, which probably has nothing to do with the node having been previously removenoded. Stop the node, wipe all the data on the node, inc

Issue with removing a node and adding it back

2015-03-26 Thread Shiwen Cheng
Hi all, I encountered an issue by removing and adding back a node. Here is how this issue came out: (1) We have four nodes cluster running, but there was a hard disk failure on one of the node. Since we need to replace the hard disk, I chose to use *removenode *to remove the failed node. (2) few d

Re: Removing a node in cluster

2012-06-06 Thread aaron morton
It depends on what you mean by remove (background info here http://www.datastax.com/docs/1.0/operations/cluster_management ) If you use nodetool decomission or nodetool removetoken the data will be redistributed. Cheers - Aaron Morton Freelance Developer @aaronmorton http://

Removing a node in cluster

2012-06-05 Thread Prakrati Agrawal
Dear all I am trying to check the performance of Cassandra on adding or removing nodes. I want to know what happens to my existing data if I remove a node ? Please help me Thanks and Regards Prakrati Agrawal | Developer - Big Data(I&D)| 9731648376 | www.mu-sigma.com _

RE: Completely removing a node from the cluster

2011-08-23 Thread Bryce Godfrey
: Brandon Williams [mailto:dri...@gmail.com] Sent: Tuesday, August 23, 2011 10:42 AM To: user@cassandra.apache.org Subject: Re: Completely removing a node from the cluster On Tue, Aug 23, 2011 at 2:26 AM, aaron morton wrote: > I'm running low on ideas for this one. Anyone else ? > >

Re: Completely removing a node from the cluster

2011-08-23 Thread Brandon Williams
On Tue, Aug 23, 2011 at 2:26 AM, aaron morton wrote: > I'm running low on ideas for this one. Anyone else ? > > If the phantom node is not listed in the ring, other nodes should not be > storing hints for it. You can see what nodes they are storing hints for via > JConsole. I think I found it i

Re: Completely removing a node from the cluster

2011-08-23 Thread aaron morton
disabled >>> Compacted row minimum size: 2816160 >>> Compacted row maximum size: 386857368 >>> Compacted row mean size: 120432714 >>> >>> Is there a way for me to manually remove this dead node? &g

Re: Completely removing a node from the cluster

2011-08-23 Thread Jonathan Colby
4 >> Row cache: disabled >> Compacted row minimum size: 2816160 >> Compacted row maximum size: 386857368 >> Compacted row mean size: 120432714 >> >> Is there a way for me to manually remove this de

Re: Completely removing a node from the cluster

2011-08-23 Thread aaron morton
or me to manually remove this dead node? > > -Original Message- > From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] > Sent: Sunday, August 21, 2011 9:09 PM > To: user@cassandra.apache.org > Subject: RE: Completely removing a node from the cluster > > It&#x

RE: Completely removing a node from the cluster

2011-08-22 Thread Bryce Godfrey
[mailto:bryce.godf...@azaleos.com] Sent: Sunday, August 21, 2011 9:09 PM To: user@cassandra.apache.org Subject: RE: Completely removing a node from the cluster It's been at least 4 days now. -Original Message- From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Sunday, August 21, 2011

RE: Completely removing a node from the cluster

2011-08-21 Thread Bryce Godfrey
It's been at least 4 days now. -Original Message- From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Sunday, August 21, 2011 3:16 PM To: user@cassandra.apache.org Subject: Re: Completely removing a node from the cluster I see the mistake I made about ring, gets the endpoint

Re: Completely removing a node from the cluster

2011-08-21 Thread aaron morton
n morton [mailto:aa...@thelastpickle.com] > Sent: Sunday, August 21, 2011 4:23 AM > To: user@cassandra.apache.org > Subject: Re: Completely removing a node from the cluster > > Unreachable nodes in either did not respond to the message or were known to > be down and were not sent a message.

RE: Completely removing a node from the cluster

2011-08-21 Thread Bryce Godfrey
[mailto:aa...@thelastpickle.com] Sent: Sunday, August 21, 2011 4:23 AM To: user@cassandra.apache.org Subject: Re: Completely removing a node from the cluster Unreachable nodes in either did not respond to the message or were known to be down and were not sent a message. The way the node lists are

Re: Completely removing a node from the cluster

2011-08-21 Thread aaron morton
Unreachable nodes in either did not respond to the message or were known to be down and were not sent a message. The way the node lists are obtained for the ring command and describe cluster are the same. So it's a bit odd. Can you connect to JMX and have a look at the o.a.c.db.StorageService

Completely removing a node from the cluster

2011-08-19 Thread Bryce Godfrey
I'm on 0.8.4 I have removed a dead node from the cluster using nodetool removetoken command, and moved one of the remaining nodes to rebalance the tokens. Everything looks fine when I run nodetool ring now, as it only lists the remaining 2 nodes and they both look fine, owning 50% of the token

Re: Removing a node ...

2011-03-09 Thread Sasha Dolgy
nice .. thanks Jonathan for the quick feedback. suppose then, for the time being, until the fix for 2283 is applied i guess i'll live with no decommission and my dirty way of removing a node. -sd On Wed, Mar 9, 2011 at 10:24 PM, Jonathan Ellis wrote: > On Wed, Mar 9, 2011 at 3:11 P

Re: Removing a node ...

2011-03-09 Thread Jonathan Ellis
On Wed, Mar 9, 2011 at 3:11 PM, Sasha Dolgy wrote: > Ok ... I have been very hesitant to upgrade from 0.7.0 because I > haven't really had many problems and the comments on 0.7.1 and 0.7.2 > weren't that encouraging . 0.7.0 is worse. > So testingon 10.0.0.1 I set up 0.7.3 and have it aut

Re: Removing a node ...

2011-03-09 Thread Sasha Dolgy
Ok ... I have been very hesitant to upgrade from 0.7.0 because I haven't really had many problems and the comments on 0.7.1 and 0.7.2 weren't that encouraging . So testingon 10.0.0.1 I set up 0.7.3 and have it auto-bootstrap: 10.0.0.3 Up Normal 229.26 KB 40.78% 240530881901956

Re: Removing a node ...

2011-03-09 Thread Jonathan Ellis
I think 2072 is something different but you should definitely upgrade before troubleshooting further. On Wed, Mar 9, 2011 at 2:00 PM, Sasha Dolgy wrote: > Hi, > > Checked logs on node where decommission command was performed and on > other nodes, and no error messages.  Just info messages.  Altho

Re: removing a node

2011-03-09 Thread aaron morton
yes. dead normally means machine cannot be started. Aaron On 10/03/2011, at 6:01 AM, Sasha Dolgy wrote: > > Hi there, > > Wanted to clarify with anyone ... re: > http://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely > > You can take a node out of the cluster with nodetool dec

Re: Removing a node ...

2011-03-09 Thread Sasha Dolgy
Hi, Checked logs on node where decommission command was performed and on other nodes, and no error messages. Just info messages. Although the behaviour and circumstances are exact, I'm wondering if https://issues.apache.org/jira/browse/CASSANDRA-2072 has something to do with it. Again, this is

Re: Removing a node ...

2011-03-09 Thread Jonathan Ellis
Did you check log for errors? (ASF mailing lists prefer plain text; html has a higher spam score.) On Wed, Mar 9, 2011 at 11:59 AM, Sasha Dolgy wrote: > mail servers keep catching up in spam filters.  Something about being a > loyal gmail user ...! > Let's try this again: > > further to this ...

Removing a node ...

2011-03-09 Thread Sasha Dolgy
mail servers keep catching up in spam filters. Something about being a loyal gmail user ...! Let's try this again: further to this ...using cassandra 0.7.0 nodetool -h 10.0.0.1 -p 8080 decommission INFO [RMI TCP Connection(2)-10.0.0.1] 2011-03-09 16:52:22,226 StorageService.java (line 399) Lea

removing a node

2011-03-09 Thread Sasha Dolgy
Hi there, Wanted to clarify with anyone ... re: http://wiki.apache.org/cassandra/Operations#Removing_nodes_entirely You can take a node out of the cluster with nodetool decommission to a live node, or nodetool removetoken (to any other machine) to remove a dead one. This will assign the ranges th