On 5/14/2020 7:22 AM, Ryan W wrote:
I manage a site where solr has stopped running a couple times in the past
week. The server hasn't been rebooted, so that's not the reason. What else
causes solr to stop running? How can I investigate why this is happening?
Any situation where Solr stops running and nobody requested the stop is
a result of a serious problem that must be thoroughly investigated. I
think it's a bad idea for Solr to automatically restart when it stops
unexpectedly. Chances are that whatever caused the crash is going to
simply make the crash happen again until the problem is solved.
Automatically restarting could hide problems from the system administrator.
The only way a Solr auto-restart would be acceptable to me is if it
sends a high priority alert to the sysadmin EVERY time it executes an
auto-restart. It really is that bad of a problem.
The causes of Solr crashes (that I can think of) include the following.
I believe I have listed these four options from most likely to least likely:
* Java OutOfMemoryError exceptions. On non-windows systems, the
"bin/solr" script starts Solr with an option that results in Solr's
death anytime one of these exceptions occurs. We do this because
program operation is indeterminate and completely unpredictable when
OOME occurs, so it's far safer to stop running. That exception can be
caused by several things, some of which actually do not involve memory
at all. If you're running on Windows via the bin\solr.cmd command, then
this will not happen ... but OOME could still cause a crash, because as
I already mentioned, program operation is unpredictable when OOME occurs.
* The OS kills Solr because system memory is completely exhausted and
Solr is the process using the most memory. Linux calls this the
"oom-killer" ... I am pretty sure something like it exists on most
operating systems.
* Corruption somewhere in the system. Could be in Java, the OS, Solr,
or data used by any of those.
* A very serious bug in Solr's code that we haven't discovered yet.
I included that last one simply for completeness. A bug that causes a
crash *COULD* exist, but as of right now, we have not seen any
supporting evidence.
My guess is that Java OutOfMemoryError is the cause here, but I can't be
certain. If that is happening, then some resource (which might not be
memory) is fully depleted. We would need to see the full
OutOfMemoryError exception in order to determine why it is happening.
Sometimes the exception is logged in solr.log, sometimes it isn't. We
cannot predict what part of the code will be running when OOME occurs,
so it would be nearly impossible for us to guarantee logging. OOME can
happen ANYWHERE - even in code that the compiler thinks is immune to
exceptions.
Side note to fellow committers: I wonder if we should implement an
uncaught exception handler in Solr. I have found in my own programs
that it helps figure out thorny problems. And while I am on the subject
of handlers that might not be general knowledge, I didn't find a
shutdown hook or a security manager outside of tests.
Thanks,
Shawn
