If you're on 3.9 it's likely unrelated as streaming_socket_timeout_in_ms is 48 hours. Appears rebuild is trying to stream the same file twice. Are there other exceptions in the logs related to the file, or can you find out if it's previously been sent by the same session? Search the logs for the file that failed and post back any exceptions.
On 29 December 2017 at 10:18, Martin Mačura <m.mac...@gmail.com> wrote: > Is this something that can be resolved by CASSANDRA-11841 ? > > Thanks, > > Martin > > On Thu, Dec 21, 2017 at 3:02 PM, Martin Mačura <m.mac...@gmail.com> wrote: > > Hi all, > > we are trying to add a new datacenter to the existing cluster, but the > > 'nodetool rebuild' command always fails after a couple of hours. > > > > We're on Cassandra 3.9. > > > > Example 1: > > > > 172.24.16.169 INFO [STREAM-IN-/172.25.16.125:55735] 2017-12-13 > > 23:55:38,840 StreamResultFuture.java:174 - [Stream > > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed. > > Receiving 0 files(0.000KiB), sending 9844 files(885.587GiB) > > 172.25.16.125 INFO [STREAM-IN-/172.24.16.169:7000] 2017-12-13 > > 23:55:38,858 StreamResultFuture.java:174 - [Stream > > #b8faf130-e092-11e7-bab5-0d4fb7c90e72 ID#0] Prepare completed. > > Receiving 9844 files(885.587GiB), sending 0 files(0.000KiB) > > > > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:55735] 2017-12-14 > > 04:28:09,064 StreamSession.java:533 - [Stream > > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on > > session with peer 172.25.16.125 > > 172.24.16.169 java.io.IOException: Connection reset by peer > > > > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:49412] 2017-12-14 > > 07:26:26,832 StreamSession.java:533 - [Stream > > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on > > session with peer 172.25.16.125 > > 172.24.16.169 java.lang.RuntimeException: Transfer of file > > <redacted>-13d700008e3f11e6a6cbe1698349da4d/mc-8659-big-Data.db > > already completed or aborted (perhaps session failed?). > > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-14 > > 07:26:50,004 StreamSession.java:533 - [Stream > > #b8faf130-e092-11e7-bab5-0d4fb7c90e72] Streaming error occurred on > > session with peer 172.24.16.169 > > 172.25.16.125 java.io.IOException: Connection reset by peer > > > > Example 2: > > > > 172.24.16.169 INFO [STREAM-IN-/172.25.16.125:35202] 2017-12-18 > > 03:24:31,423 StreamResultFuture.java:174 - [Stream > > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed. > > Receiving 0 files(0.000KiB), sending 12312 files(895.973GiB) > > 172.25.16.125 INFO [STREAM-IN-/172.24.16.169:7000] 2017-12-18 > > 03:24:31,441 StreamResultFuture.java:174 - [Stream > > #95d36300-e3d4-11e7-a90b-2b89506ad2af ID#0] Prepare completed. > > Receiving 12312 files(895.973GiB), sending 0 files(0.000KiB) > > > > 172.24.16.169 ERROR [STREAM-IN-/172.25.16.125:35202] 2017-12-18 > > 06:39:42,049 StreamSession.java:533 - [Stream > > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on > > session with peer 172.25.16.125 > > 172.24.16.169 java.io.IOException: Connection reset by peer > > > > 172.24.16.169 ERROR [STREAM-OUT-/172.25.16.125:42744] 2017-12-18 > > 09:25:36,188 StreamSession.java:533 - [Stream > > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on > > session with peer 172.25.16.125 > > 172.24.16.169 java.lang.RuntimeException: Transfer of file > > <redacted>-3b5782d08e4411e6842917253f111990/mc-152979-big-Data.db > > already completed or aborted (perhaps session failed?). > > 172.25.16.125 ERROR [STREAM-OUT-/172.24.16.169:7000] 2017-12-18 > > 09:25:59,447 StreamSession.java:533 - [Stream > > #95d36300-e3d4-11e7-a90b-2b89506ad2af] Streaming error occurred on > > session with peer 172.24.16.169 > > 172.25.16.125 java.io.IOException: Connection timed out > > > > Datacenter: PRIMARY > > =================== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns (effective) Host ID > > Rack > > UN 172.24.16.169 918.31 GiB 256 100.0% > > bc4a980b-cca6-4ca2-b32f-f8206d48e14c RAC1 > > UN 172.24.16.170 908.76 GiB 256 100.0% > > 37b2742e-c83a-4341-896f-09d244810e69 RAC1 > > UN 172.24.16.171 908.44 GiB 256 100.0% > > 6dc2b9d8-75dd-48f8-858c-53b1af42e8fb RAC1 > > Datacenter: SECONDARY > > ===================== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns (effective) Host ID > > Rack > > UN 172.25.16.125 27.48 GiB 256 100.0% > > 1e1669eb-cfd2-4718-a073-558946a8c947 RAC2 > > UN 172.25.16.124 28.24 GiB 256 100.0% > > 896d9894-10c8-4269-9476-5ddab3c8abe9 RAC2 > > > > Any ideas? > > > > Thanks, > > > > Martin > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
