Re: Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
Agree with you on that. Just wanted to highlight that I am experiencing the same behavior. Regards Manish On Tue, Jan 18, 2022, 22:50 Bowen Song wrote: > The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra > was full of bugs at that time, and it has improved a lot since the

Re: Migration between Apache 4.x and DSE 6+?

2022-01-18 Thread Erick Ramirez
DSE 6.x is compatible with C* 3.11. In any case, there are a lot of sharp edges with mixing OSS C* and DSE nodes so it's not recommended. It is going to be addressed in a future release. Cheers!

Re: Hanging repairs in Cassandra

2022-01-18 Thread Bowen Song
The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra was full of bugs at that time, and it has improved a lot since then. For that reason, I would rather not compare the issue you have with some 9 years old issues someone else had. On 18/01/2022 16:11, manish khandelwal wr

Re: Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
I am not sure what is happening but it has happened thrice. It is happening that merkle trees are not received from nodes of other data center. Getting issue on similar lines as mentioned here https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledg

Migration between Apache 4.x and DSE 6+?

2022-01-18 Thread Durity, Sean R
Has anyone been able to add Apache Cassandra 4.x nodes to a new DC within a DSE 6+ cluster (or vice versa) in order to migrate from one to the other with no downtime? I was able to do this prior to DSE 6/Cassandra 4.0, but that was before the internals rewrite (and different sstable format?) of

Re: Hanging repairs in Cassandra

2022-01-18 Thread Bowen Song
Keep reading the log on the initiator and the node sending the merkle tree, anything follows that? FYI, not all log has the repair ID in it, therefore please read the relevant logs in the chronological order without filtering (e.g. "grep") on the repair ID. I'm sceptical network issue is causi

Re: Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
In the system logs, on the node where repair was initiated, I see that the node has requested merkle tree from all nodes including itself INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for *tablename* (to [*/

Re: Hanging repairs in Cassandra

2022-01-18 Thread Bowen Song
The entry in the debug.log is not specific to a repair session, and it could also be caused by reasons other than network connectivity issue, such as long STW GC pauses. I usually don't start troubleshooting an issue from the debug log, as it can be rather noisy. The system.log is a better star

Hanging repairs in Cassandra

2022-01-18 Thread manish khandelwal
I have a Cassandra 3.11.2 cluster with two DCs. While running repair , I am observing the following behavior. I am seeing that node is not able to receive merkle tree from one or two nodes. Also I am able to see that the missing nodes did send the merkle tree but it was not received. This make rep