If you have an SNMP infrastructure available (nagios or similar) you
should be able to set up a polling monitor that will keep statistics on
the number of threads in your jvm and even allow you to inspect their
stacks remotely. You can set alarms so you will be notified if cpu
thread count or other metrics exceed a configurable threshold and then
take a look at the process before it goes off the deep end. It is a
fair amount of work to set this up, but really useful if you need to
support a critical system.
-Mike
On 6/1/2011 3:42 PM, Jonathan Rochkind wrote:
First guess (and it really is just a guess) would be Java garbage
collection taking over. There are some JVM parameters you can use to
tune the GC process, especially if the machine is multi-core, making
sure GC happens in a seperate thread is helpful.
But figuring out exactly what's going on requires confusing JVM
debugging of which I am no expert at either.
On 6/1/2011 3:04 PM, Chris Cowan wrote:
About once a day a Solr/Jetty process gets hung on my server
consuming 100% of one of the CPU's. Once this happens the server no
longer responds to requests. I've looked through the logs to try and
see if anything stands out but so far I've found nothing out of the
ordinary.
My current remedy is to log in and just kill the single processes
that's hung. Once that happens everything goes back to normal and I'm
good for a day or so. I'm currently the running following:
solr-jetty-1.4.0+ds1-1ubuntu1
which is comprised of
Solr 1.4.0
Jetty 6.1.22
on Unbuntu 10.10
I'm pretty new to managing a Jetty/Solr instance so at this point I'm
just looking for advice on how I should go about trouble shooting
this problem.
Chris
