Re: Node stuck during nodetool rebuild

2014-08-06 Thread Vasileios Vlachos
Actually something else I would like to ask... Do you know if phi is related to streaming_socket_timeout_in_ms? It seems to be set to infinity by default. Could that be related to the hang behaviour of rebuild? Would you recommend changing the default or I have completely misinterpreted its meaning

Re: Node stuck during nodetool rebuild

2014-08-06 Thread Vasileios Vlachos
Hello Mark and Rob, Thank you very much for your input, I will increase the phi threshold and report back any progress. Vasilis On 5 Aug 2014 21:52, "Mark Reddy" wrote: > Hi Vasilis, > > To further on what Rob said > > I believe you might be able to tune the phi detector threshold to help >> th

Re: Node stuck during nodetool rebuild

2014-08-05 Thread Mark Reddy
Hi Vasilis, To further on what Rob said I believe you might be able to tune the phi detector threshold to help this > operation complete, hopefully someone with direct experience of same will > chime in. I have been through this operation where streams break due to a node falsely being marked d

Re: Node stuck during nodetool rebuild

2014-08-05 Thread Robert Coli
On Tue, Aug 5, 2014 at 1:28 AM, Vasileios Vlachos < vasileiosvlac...@gmail.com> wrote: > The problem is that the nodetool seems to be stuck, and nodetool netstats > on node1 of DC2 appears to be stuck at 10% streaming a 5G file from node2 > at DC1. This doesn't tally with nodetool netstats when ru

Node stuck during nodetool rebuild

2014-08-05 Thread Vasileios Vlachos
Hello All, We are on 1.2.18 (running on Ubuntu 12.04) and we recently tried to add a second DC on our demo environment, just before trying it on live. The existing DC1 has two nodes which approximately hold 10G of data (RF=2). In order to add the second DC, DC2, we followed this procedure: On DC1