Re: System hints compaction stuck

2013-08-07 Thread David McNelis
Fwiw, similar to another issue of stuck compaction that was on the list several days ago, if I cleared out the hints, either by removing files while node was down, or running a scrub on system.hints during node startup, I was able to get these compactions cleared, an the nodes are starting to get c

Re: System hints compaction stuck

2013-08-07 Thread Nate McCall
Is there anything else on the network that could be attempting to connect to 9160? That is the exact error you would get when someone initiates a connection and sends a null byte. You can reproduce it thusly: echo -n 'm' | nc localhost 9160 On Wed, Aug 7, 2013 at 11:11 AM, David McNelis wrote:

Re: System hints compaction stuck

2013-08-07 Thread David McNelis
Nate, We had a node that was flaking on us last week and had a lot of handoffs fail to that node. We ended up decommissioning that node entirely. I can't find the actual error we were getting at the time (logs have been rotated out), but currently we're not seeing any errors there. We haven't h

Re: System hints compaction stuck

2013-08-07 Thread Nate McCall
Thrift and ClientState are both unrelated to hints. What do you see in the logs after "Started hinted handoff for host:..." from HintedHandoffManager? It should either have an error message or something along the lines of "Finished hinted handoff of:..." Where there any schema updates that prece

System hints compaction stuck

2013-08-07 Thread David McNelis
Morning folks, For the last couple of days all of my nodes (17, all running 1.2.8) have been stuck at various percentages of completion for compacting system.hints. I've tried restarting the nodes (including a full rolling restart of the cluster) to no avail. When I turn on Debugging I am seeing