Looks like the connecting to ZooKeeper is "flapping". So as it tries to recover it keeps losing the connection to zookeeper and then trying again, and I don't have enough of the log to tell, but that probably just repeats and repeats.
I guess the network is probably not so fast and or the load and network are not working out so well… Try raising the zkClientTimeout - see the FAQ: http://wiki.apache.org/solr/SolrCloud#FAQ It defaults to like 15 seconds. You might try 30 or 40 seconds. If that doesn't help, you have to figure out why Solr is having so much trouble communicating with ZooKeeper. Are you using an external ensemble, a single ZooKeeper node, embedded ZooKeeper? - Mark On Dec 5, 2012, at 7:59 PM, Nathaniel Domingo <niel.domi...@gmail.com> wrote: > here's a link to a portion of the log in pastebin. > > http://pastebin.com/UDBMDdMv > > Thanks > > > On Thu, Dec 6, 2012 at 11:53 AM, Mark Miller <markrmil...@gmail.com> wrote: > >> I think the list strips most attachments or something - can you try >> something like pastebin.com? >> >> Thanks, >> >> mark >> >> On Dec 5, 2012, at 7:46 PM, Nathaniel Domingo <niel.domi...@gmail.com> >> wrote: >> >>> attached is a log relevant to the recovery. >>> >>> Thanks >>> >>> >>> On Thu, Dec 6, 2012 at 11:23 AM, Mark Miller <markrmil...@gmail.com> >> wrote: >>> Okay - logs from that node would help a lot then (or just the parts >> around when it's trying to recover). >>> >>> - Mark >>> >>> On Dec 5, 2012, at 7:11 PM, Nathaniel Domingo <niel.domi...@gmail.com> >> wrote: >>> >>>> yes, i tried restarting the node twice already and both times it just >> got >>>> stucked in recovering. one node also had some problems a few days ago, >>>> after a restart, it eventually moved from recovering to active after an >>>> hour. i'm using solr 4.0.0. >>>> >>>> Thanks >>>> >>>> >>>> On Thu, Dec 6, 2012 at 11:03 AM, Mark Miller <markrmil...@gmail.com> >> wrote: >>>> >>>>> Did you try restarting that node? Have you seen a successful recovery >>>>> before? What exact version are you using? >>>>> >>>>> Can you share any related info in the logs for that node? >>>>> >>>>> - Mark >>>>> >>>>> On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <niel.domi...@gmail.com >>> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm very new to solr, less than a month. I just set-up a solrcloud >>>>> cluster >>>>>> last week and has encountered a problem. i have four nodes with two >>>>> shards. >>>>>> one node is stuck at recovering for days now. how do i go about >> fixing >>>>> this? >>>>>> >>>>>> Thanks >>>>> >>>>> >>> >>> >> >>
