Re: Debugging a Solr/Jetty Hung Process

2011-06-07 Thread Chris Cowan
OK... The fix I thought would fix it didn't fix it (which was to use the commitWithin feature). What I can gather from `ps` is that the thread has pages locked in memory. Currently I'm using native locking for Solr. Would switching to simple help alleviate this problem? Chris On Jun 4, 2011, a

Re: Debugging a Solr/Jetty Hung Process

2011-06-04 Thread Chris Cowan
oops... http://search.lucidimagination.com/search/document/bf43af733e898424/busywait_hang_using_extracting_update_handler_on_trunk Chris On Jun 4, 2011, at 2:48 PM, Chris Cowan wrote: > I found this thread that looks similar to what's happening on my system. I > think what happens is there ar

Re: Debugging a Solr/Jetty Hung Process

2011-06-04 Thread Chris Cowan
I found this thread that looks similar to what's happening on my system. I think what happens is there are multiple commits happening at once from the clients and it's causing the same issue. I'm going to use the commitWithin argument to the updates to see if that fixes the problem. I will repor

Re: Debugging a Solr/Jetty Hung Process

2011-06-02 Thread Michael Sokolov
If you have an SNMP infrastructure available (nagios or similar) you should be able to set up a polling monitor that will keep statistics on the number of threads in your jvm and even allow you to inspect their stacks remotely. You can set alarms so you will be notified if cpu thread count or

Re: Debugging a Solr/Jetty Hung Process

2011-06-01 Thread Jonathan Rochkind
First guess (and it really is just a guess) would be Java garbage collection taking over. There are some JVM parameters you can use to tune the GC process, especially if the machine is multi-core, making sure GC happens in a seperate thread is helpful. But figuring out exactly what's going on

Re: Debugging a Solr/Jetty Hung Process

2011-06-01 Thread Chris Cowan
Sorry ... I just found it. I will try that next time. I have a feeling it wont work since the server usually stops accepting connections. Chris On Jun 1, 2011, at 12:12 PM, Chris Cowan wrote: > I'm pretty green... is that something I can do while the event is happening > or is there something

Re: Debugging a Solr/Jetty Hung Process

2011-06-01 Thread Chris Cowan
I'm pretty green... is that something I can do while the event is happening or is there something I need to configure to capture the dump ahead of time. I've tried to reproduce the problem by putting the server under load but that doesn't seem to be the issue. Chris On Jun 1, 2011, at 12:06

Re: Debugging a Solr/Jetty Hung Process

2011-06-01 Thread Bill Au
Taking a thread dump will take you what's going. Bill On Wed, Jun 1, 2011 at 3:04 PM, Chris Cowan wrote: > About once a day a Solr/Jetty process gets hung on my server consuming 100% > of one of the CPU's. Once this happens the server no longer responds to > requests. I've looked through the log

Debugging a Solr/Jetty Hung Process

2011-06-01 Thread Chris Cowan
About once a day a Solr/Jetty process gets hung on my server consuming 100% of one of the CPU's. Once this happens the server no longer responds to requests. I've looked through the logs to try and see if anything stands out but so far I've found nothing out of the ordinary. My current remedy