Re: Bootstraping is failing

2020-05-11 Thread Reid Pinchback
it seems like it could be a case of refining behavior on the periphery of the problem, not anything directly addressing it. From: Surbhi Gupta Reply-To: "user@cassandra.apache.org" Date: Saturday, May 9, 2020 at 11:51 AM To: "user@cassandra.apache.org" Subject: Re:

Re: Bootstraping is failing

2020-05-09 Thread Surbhi Gupta
I tried to change the heap size from 31GB to 62GB on the bootstrapping node because , I noticed that , when it reached the mid way of bootstrapping , heap reached to around 90% or more and node just freeze . But still it is the same behavior , it again reached midway and heap again reached 90% or m

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
When we are starting the node, it is starting bootstrap automatically and restreaming the whole data again. It is not resuming . On Thu, May 7, 2020 at 4:47 PM Adam Scott wrote: > I think you want to run `nodetool bootstrap resume` ( > https://cassandra.apache.org/doc/latest/tools/nodetool/boot

Re: Bootstraping is failing

2020-05-07 Thread Adam Scott
I think you want to run `nodetool bootstrap resume` ( https://cassandra.apache.org/doc/latest/tools/nodetool/bootstrap.html) to pick up where it last left off. Sorry for the late reply. On Thu, May 7, 2020 at 2:22 PM Surbhi Gupta wrote: > So after failed bootstrapped , if we start cassandra ag

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
So after failed bootstrapped , if we start cassandra again on the new node , will it resume bootstrap or will it start over? On Thu, 7 May 2020 at 13:32, Adam Scott wrote: > I recommend it on all nodes. This will eliminate that as a source of > trouble further on down the road. > > > On Thu, Ma

Re: Bootstraping is failing

2020-05-07 Thread Adam Scott
I recommend it on all nodes. This will eliminate that as a source of trouble further on down the road. On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta wrote: > streaming_socket_timeout_in_ms is 24 hour. > So tcp settings should be changed on the new bootstrap node or on all > nodes ? > > > On Th

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
streaming_socket_timeout_in_ms is 24 hour. So tcp settings should be changed on the new bootstrap node or on all nodes ? On Thu, 7 May 2020 at 13:23, Adam Scott wrote: > > *edit > /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 > net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10*

Re: Bootstraping is failing

2020-05-07 Thread Adam Scott
*edit /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10* then run sysctl -p to cause the kernel to reload the settings 5 minutes (300) seconds is probably too long. On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta wrote: > [root@abc cassand

RE: Bootstraping is failing

2020-05-07 Thread ZAIDI, ASAD
heck if [streaming_socket_timeout_in_ms ] setting in Cassandra.yaml file if that sufficient enough before streaming is interrupted ? ~Asad From: Surbhi Gupta [mailto:surbhi.gupt...@gmail.com] Sent: Thursday, May 7, 2020 3:09 PM To: user@cassandra.apache.org Subject: Re: Bootstraping is

Re: Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
[root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time 300 [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl 30 [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes 9 On Thu, 7 May 2020 at 12:32, Adam Scott wrote: > Maybe a firewall killing a connection?

Re: Bootstraping is failing

2020-05-07 Thread Adam Scott
Maybe a firewall killing a connection? What does the following show? cat /proc/sys/net/ipv4/tcp_keepalive_time cat /proc/sys/net/ipv4/tcp_keepalive_intvl cat /proc/sys/net/ipv4/tcp_keepalive_probes On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta wrote: > Hi, > > We are trying to expand a datacente

Bootstraping is failing

2020-05-07 Thread Surbhi Gupta
Hi, We are trying to expand a datacenter and trying to add nodes but when node is bootstrapping , it goes half way through and then fail with below error, We have increased stremthroughput from 200 to 400 when we were trying for the 2nd time but still it failed. We are on 3.11.0 , using G1GC with