You can detect your node in distress by using different tools like Monit ( https://mmonit.com/monit/ open source) which alerts by email when disk space utilization goes above a threshold or memory usage goes above threshold etc. You can also have Monit to restart Solr process in some of those situations etc.
Wiley is another commercial solution. Thanks, Susheel On Mon, Dec 19, 2016 at 9:07 PM, Erick Erickson <erickerick...@gmail.com> wrote: > The first question is _why_ is your disk full? Older versions of Solr > could, for instance, accumulate solr console log files forever. If > that's the case, just stop the Solr instance on that node, remove the > Solr log files, fix the log4j.properties file to not append to console > forever and you're done. > > If it's more complicated, I'd > 1> Spin up a new Solr node with adequate disk space. > 2> use the Collections API ADDREPLICA command to add new replicas to > this node. The indexes will be synchronized etc. > 3> once <2> is complete and the newly added replicas are green, issue > a DELETEREPLICA on the replicas on the Solr node in distress. > > You could issue the DELETEREPLICA before creating a new Solr instance > on a new node, but doing it in the order I indicated has the fewest > chances for doing something awful like deleting the replica on the > good node by mistake without having a healthy replica in your > cluster.... > > Best, > Erick > > On Mon, Dec 19, 2016 at 5:56 PM, Brian Narsi <bnars...@gmail.com> wrote: > > We have had a situation where Solr node was in distress due to hard drive > > being full and the queries became very slow. Since our Solr cluster has > two > > nodes with indexes being fully available on both the nodes, we think that > > one good solution would be to just stop the Solr instance on a distressed > > node. That way the other node will continue to serve queries until the > > distressed node is fixed. > > > > Can folks please share their experiences in such a situation? > > > > What are some good ways to handle this? How/when should we detect that > the > > host (linux) is in distress and then stop the solr node? > > > > Thanks, >
