Re: Stalled Bootstrapping Process

2010-04-02 Thread Chris Goffinet
+1 On Fri, Apr 2, 2010 at 3:49 PM, Jonathan Ellis wrote: > Ah, right. That's confusing for everyone. I think the best solution > there is to just get > http://issues.apache.org/jira/browse/CASSANDRA-579 done so it can > start streaming immediately. > > On Fri, Apr 2, 2010 at 5:45 PM, Dan Di Sp

Re: Stalled Bootstrapping Process

2010-04-02 Thread Dan Di Spaltro
I agree. That would have other good side-effects, like minimizing shooting yourself in the foot, for new folks. On Fri, Apr 2, 2010 at 3:49 PM, Jonathan Ellis wrote: > Ah, right.  That's confusing for everyone.  I think the best solution > there is to just get > http://issues.apache.org/jira/bro

Re: Stalled Bootstrapping Process

2010-04-02 Thread Jonathan Ellis
Ah, right. That's confusing for everyone. I think the best solution there is to just get http://issues.apache.org/jira/browse/CASSANDRA-579 done so it can start streaming immediately. On Fri, Apr 2, 2010 at 5:45 PM, Dan Di Spaltro wrote: > It did once it was actually done anti-compacting.  The

Re: Stalled Bootstrapping Process

2010-04-02 Thread Dan Di Spaltro
It did once it was actually done anti-compacting. The biggest question-mark (for us) was, what was happening during the anti-compaction phase. On Fri, Apr 2, 2010 at 3:39 PM, Jonathan Ellis wrote: > Great, glad it worked. > > Sounds like we do have a bug though if the destination node never > sh

Re: Stalled Bootstrapping Process

2010-04-02 Thread Jonathan Ellis
Great, glad it worked. Sounds like we do have a bug though if the destination node never showed anything in Streaming mbean. :( On Fri, Apr 2, 2010 at 5:11 PM, Dan Di Spaltro wrote: > To close the loop on this, the node finished bootstrapping.  The > source node rebooting definitely halted the p

Re: Stalled Bootstrapping Process

2010-04-02 Thread Dan Di Spaltro
To close the loop on this, the node finished bootstrapping. The source node rebooting definitely halted the process. Visibility-wise, watching the anti-compactions is the best way to tell how much progress is being made on the bootstrapping process. The CompactionManager mbean gives you insight

Re: Stalled Bootstrapping Process

2010-04-01 Thread Jonathan Ellis
I would turn debug logging on globally on the new node, that will answer more questions than just the streaming package.

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
Seems to be doing more stuff now. Ive attached an updated screenshot. On Thu, Apr 1, 2010 at 1:16 PM, Jonathan Ellis wrote: > Right. > > On Thu, Apr 1, 2010 at 3:15 PM, Dan Di Spaltro > wrote: >> So it looks like its still performing anti-compaction.  The >> compactionmanager is the best way t

Re: Stalled Bootstrapping Process

2010-04-01 Thread Jonathan Ellis
Right. On Thu, Apr 1, 2010 at 3:15 PM, Dan Di Spaltro wrote: > So it looks like its still performing anti-compaction.  The > compactionmanager is the best way to track this? > > On Thu, Apr 1, 2010 at 12:31 PM, Dan Di Spaltro > wrote: >> Sorry I meant the red one restarted about a day ago.  The

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
So it looks like its still performing anti-compaction. The compactionmanager is the best way to track this? On Thu, Apr 1, 2010 at 12:31 PM, Dan Di Spaltro wrote: > Sorry I meant the red one restarted about a day ago.  The graph shows > the dip in disk space.  But it no where near returned to th

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
Sorry I meant the red one restarted about a day ago.  The graph shows the dip in disk space.  But it no where near returned to the previous amount of disk usage.  I was referring to how the red one didn't reclaim all its space (I figure about 60gb actually belong on that machine) Is that normal (it

Re: Stalled Bootstrapping Process

2010-04-01 Thread Jonathan Ellis
On Thu, Apr 1, 2010 at 2:22 PM, Dan Di Spaltro wrote: > But I didn't restart the red one. >> >> > On Thu, Apr 1, 2010 at 11:57 AM, Dan Di Spaltro >> >> > >> >> > wrote: >> >> >> >> >> >> Red one. >> >> >> >> >> >> On Thu, Apr 1, 2010 at 11:55 AM, Jonathan Ellis >> >> >> wrote: >> >> >>> >> >> >

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
But I didn't restart the red one. On Thu, Apr 1, 2010 at 12:18 PM, Jonathan Ellis wrote: > There shouldn't be anything to clean up. (The temporary streaming > files it anticompacted are automatically removed on restart) > > On Thu, Apr 1, 2010 at 2:17 PM, Dan Di Spaltro > wrote: > > Okay, so s

Re: Stalled Bootstrapping Process

2010-04-01 Thread Jonathan Ellis
There shouldn't be anything to clean up. (The temporary streaming files it anticompacted are automatically removed on restart) On Thu, Apr 1, 2010 at 2:17 PM, Dan Di Spaltro wrote: > Okay, so should I run any more commands like cleanup before? > > On Thu, Apr 1, 2010 at 12:09 PM, Jonathan Ellis

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
Okay, so should I run any more commands like cleanup before? On Thu, Apr 1, 2010 at 12:09 PM, Jonathan Ellis wrote: > Bootstrap source restarting will always fail bootstrap. You'll need > to restart the blue one too now, I'm afraid. > > On Thu, Apr 1, 2010 at 2:01 PM, Dan Di Spaltro > wrote: >

Re: Stalled Bootstrapping Process

2010-04-01 Thread Jonathan Ellis
Bootstrap source restarting will always fail bootstrap. You'll need to restart the blue one too now, I'm afraid. On Thu, Apr 1, 2010 at 2:01 PM, Dan Di Spaltro wrote: > Before the Red one rebooted it had 1 active STREAM-STAGE.  Now it has 0 in > STREAM-STAGE. > > On Thu, Apr 1, 2010 at 11:57 AM,

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
Before the Red one rebooted it had 1 active STREAM-STAGE. Now it has 0 in STREAM-STAGE. On Thu, Apr 1, 2010 at 11:57 AM, Dan Di Spaltro wrote: > Red one. > > Gary - both say nothing is happening with no destinations or sources. > > > On Thu, Apr 1, 2010 at 11:55 AM, Jonathan Ellis wrote: > >> w

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
Red one. Gary - both say nothing is happening with no destinations or sources. On Thu, Apr 1, 2010 at 11:55 AM, Jonathan Ellis wrote: > which node rebooted, the red one, or the blue one? > > On Thu, Apr 1, 2010 at 11:26 AM, Dan Di Spaltro > wrote: > > So we are adding another node to the clust

Re: Stalled Bootstrapping Process

2010-04-01 Thread Jonathan Ellis
which node rebooted, the red one, or the blue one? On Thu, Apr 1, 2010 at 11:26 AM, Dan Di Spaltro wrote: > So we are adding another node to the cluster with the latest 0.6 branch > (RC1).  It seems to be hung in some limbo state. > Before bootstrapping our cluster had 50-60GB spread fairly evenl

Re: Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
The light-blue machine is in Operation Mode: Bootstrap On Thu, Apr 1, 2010 at 9:26 AM, Dan Di Spaltro wrote: > So we are adding another node to the cluster with the latest 0.6 branch > (RC1). It seems to be hung in some limbo state. > > Before bootstrapping our cluster had 50-60GB spread fairly

Re: Stalled Bootstrapping Process

2010-04-01 Thread Gary Dusbabek
Does the JMX StreamingService list any incoming/outgoing files/hosts on the sending/receiving nodes? Gary. On Thu, Apr 1, 2010 at 10:26, Dan Di Spaltro wrote: > So we are adding another node to the cluster with the latest 0.6 branch > (RC1).  It seems to be hung in some limbo state. > Before boo

Stalled Bootstrapping Process

2010-04-01 Thread Dan Di Spaltro
So we are adding another node to the cluster with the latest 0.6 branch (RC1). It seems to be hung in some limbo state. Before bootstrapping our cluster had 50-60GB spread fairly evenly across 4 machines, with RF=3. One machine had more load than the others, and sure enough bootstrapping select