We have a 16 node cluster of Solr (5.2.1) and 5 node Zookeeper (3.4.6). All the Solr nodes were registered to Zookeeper (ls /live_nodes) when setup was done 3 months back. Suddenly, few days back our search started failing because one of the solr node(consider s16) was not seen in Zookeeper, i.e., when we checked for *"ls /live_nodes"*, *s16 *solr node was not found. However, the corresponding Solr process was up and running.
To my surprise, I couldn't find any errors or warnings in solr or zookeeper logs related to this. I have few questions - 1. Is there any reason why this registration to ZK was lost? I know logs should provide some information, but, it didn't. Did anyone encountered similar issue, if so, what can be the root cause? 2. Shouldn't Solr be clever enough to detect that the registration to ZK was lost (for some reason) and should try to re-register again? PS: The issue is resolved by restarting the Solr node. However, I am curious to know why it happened in the first place. Thanks
