Thanks Erick! Should I create a JIRA issue for the same? Regarding the logs, I have changed the log level to WARN. That may be the reason, I couldn't get anything from it.
Thanks, Manohar On Tue, Dec 6, 2016 at 9:58 PM, Erick Erickson <erickerick...@gmail.com> wrote: > Most likely reason is that the Solr node in question, > was not reachable thus it was removed from > live_nodes. Perhaps due to temporary network > glitch, long GC pause or the like. If you're rolling > your logs over it's quite possible that any illuminating > messages were lost. The default 4M size for each > log is quite lo at INFO level... > > It does seem possible for a Solr node to periodically > check its status and re-insert itself into live_nodes, > go through recovery and all that. So far most of that > registration logic is baked into startup code. What > do others think? Worth a JIRA? > > Erick > > On Tue, Dec 6, 2016 at 3:53 AM, Manohar Sripada <manohar...@gmail.com> > wrote: > > We have a 16 node cluster of Solr (5.2.1) and 5 node Zookeeper (3.4.6). > > > > All the Solr nodes were registered to Zookeeper (ls /live_nodes) when > setup > > was done 3 months back. Suddenly, few days back our search started > failing > > because one of the solr node(consider s16) was not seen in Zookeeper, > i.e., > > when we checked for *"ls /live_nodes"*, *s16 *solr node was not found. > > However, the corresponding Solr process was up and running. > > > > To my surprise, I couldn't find any errors or warnings in solr or > zookeeper > > logs related to this. I have few questions - > > > > 1. Is there any reason why this registration to ZK was lost? I know logs > > should provide some information, but, it didn't. Did anyone encountered > > similar issue, if so, what can be the root cause? > > 2. Shouldn't Solr be clever enough to detect that the registration to ZK > > was lost (for some reason) and should try to re-register again? > > > > PS: The issue is resolved by restarting the Solr node. However, I am > > curious to know why it happened in the first place. > > > > Thanks >
