Debugging a Solr/Jetty Hung Process
About once a day a Solr/Jetty process gets hung on my server consuming 100% of one of the CPU's. Once this happens the server no longer responds to requests. I've looked through the logs to try and see if anything stands out but so far I've found nothing out of the ordinary. My current remedy is to log in and just kill the single processes that's hung. Once that happens everything goes back to normal and I'm good for a day or so. I'm currently the running following: solr-jetty-1.4.0+ds1-1ubuntu1 which is comprised of Solr 1.4.0 Jetty 6.1.22 on Unbuntu 10.10 I'm pretty new to managing a Jetty/Solr instance so at this point I'm just looking for advice on how I should go about trouble shooting this problem. Chris
Re: Debugging a Solr/Jetty Hung Process
I'm pretty green... is that something I can do while the event is happening or is there something I need to configure to capture the dump ahead of time. I've tried to reproduce the problem by putting the server under load but that doesn't seem to be the issue. Chris On Jun 1, 2011, at 12:06 PM, Bill Au wrote: > Taking a thread dump will take you what's going. > > Bill > > On Wed, Jun 1, 2011 at 3:04 PM, Chris Cowan > wrote: > >> About once a day a Solr/Jetty process gets hung on my server consuming 100% >> of one of the CPU's. Once this happens the server no longer responds to >> requests. I've looked through the logs to try and see if anything stands out >> but so far I've found nothing out of the ordinary. >> >> My current remedy is to log in and just kill the single processes that's >> hung. Once that happens everything goes back to normal and I'm good for a >> day or so. I'm currently the running following: >> >> solr-jetty-1.4.0+ds1-1ubuntu1 >> >> which is comprised of >> >> Solr 1.4.0 >> Jetty 6.1.22 >> on Unbuntu 10.10 >> >> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just >> looking for advice on how I should go about trouble shooting this problem. >> >> Chris
Re: Debugging a Solr/Jetty Hung Process
Sorry ... I just found it. I will try that next time. I have a feeling it wont work since the server usually stops accepting connections. Chris On Jun 1, 2011, at 12:12 PM, Chris Cowan wrote: > I'm pretty green... is that something I can do while the event is happening > or is there something I need to configure to capture the dump ahead of time. > > I've tried to reproduce the problem by putting the server under load but that > doesn't seem to be the issue. > > Chris > > On Jun 1, 2011, at 12:06 PM, Bill Au wrote: > >> Taking a thread dump will take you what's going. >> >> Bill >> >> On Wed, Jun 1, 2011 at 3:04 PM, Chris Cowan >> wrote: >> >>> About once a day a Solr/Jetty process gets hung on my server consuming 100% >>> of one of the CPU's. Once this happens the server no longer responds to >>> requests. I've looked through the logs to try and see if anything stands out >>> but so far I've found nothing out of the ordinary. >>> >>> My current remedy is to log in and just kill the single processes that's >>> hung. Once that happens everything goes back to normal and I'm good for a >>> day or so. I'm currently the running following: >>> >>> solr-jetty-1.4.0+ds1-1ubuntu1 >>> >>> which is comprised of >>> >>> Solr 1.4.0 >>> Jetty 6.1.22 >>> on Unbuntu 10.10 >>> >>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just >>> looking for advice on how I should go about trouble shooting this problem. >>> >>> Chris >
Re: Debugging a Solr/Jetty Hung Process
I found this thread that looks similar to what's happening on my system. I think what happens is there are multiple commits happening at once from the clients and it's causing the same issue. I'm going to use the commitWithin argument to the updates to see if that fixes the problem. I will report back with any findings. Chris On Jun 1, 2011, at 12:42 PM, Jonathan Rochkind wrote: > First guess (and it really is just a guess) would be Java garbage > collection taking over. There are some JVM parameters you can use to > tune the GC process, especially if the machine is multi-core, making > sure GC happens in a seperate thread is helpful. > > But figuring out exactly what's going on requires confusing JVM > debugging of which I am no expert at either. > > On 6/1/2011 3:04 PM, Chris Cowan wrote: >> About once a day a Solr/Jetty process gets hung on my server consuming 100% >> of one of the CPU's. Once this happens the server no longer responds to >> requests. I've looked through the logs to try and see if anything stands out >> but so far I've found nothing out of the ordinary. >> >> My current remedy is to log in and just kill the single processes that's >> hung. Once that happens everything goes back to normal and I'm good for a >> day or so. I'm currently the running following: >> >> solr-jetty-1.4.0+ds1-1ubuntu1 >> >> which is comprised of >> >> Solr 1.4.0 >> Jetty 6.1.22 >> on Unbuntu 10.10 >> >> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just >> looking for advice on how I should go about trouble shooting this problem. >> >> Chris
Re: URGENT HELP: Improving Solr indexing time
How long does the query against the DB take (outside of Solr)? If that's slow then it's going to take a while to update the index. You might need to figure a way to break things up a bit, maybe use a delta import instead of a full import. Chris On Jun 4, 2011, at 6:23 AM, Rohit Gupta wrote: > My Solr server takes very long to update index. The table it hits to index is > huge with 10Million + records , but even in that case I feel this is very > long > time to index. Below is the snapshot of the /dataimport page > > busy > A command is still running... > > 1:53:39.664 > 16276 > 24237 > 16273 > 0 > 2011-06-04 11:25:26 > > > How can i determine why this is happening and how can I improve this. During > all > our test on the local server before the migration we could index 5 million > records in 4-5 hrs, but now its taking too long on the live server. > > Regards, > Rohit
Re: Debugging a Solr/Jetty Hung Process
oops... http://search.lucidimagination.com/search/document/bf43af733e898424/busywait_hang_using_extracting_update_handler_on_trunk Chris On Jun 4, 2011, at 2:48 PM, Chris Cowan wrote: > I found this thread that looks similar to what's happening on my system. I > think what happens is there are multiple commits happening at once from the > clients and it's causing the same issue. I'm going to use the commitWithin > argument to the updates to see if that fixes the problem. I will report back > with any findings. > > Chris > > On Jun 1, 2011, at 12:42 PM, Jonathan Rochkind wrote: > >> First guess (and it really is just a guess) would be Java garbage >> collection taking over. There are some JVM parameters you can use to >> tune the GC process, especially if the machine is multi-core, making >> sure GC happens in a seperate thread is helpful. >> >> But figuring out exactly what's going on requires confusing JVM >> debugging of which I am no expert at either. >> >> On 6/1/2011 3:04 PM, Chris Cowan wrote: >>> About once a day a Solr/Jetty process gets hung on my server consuming 100% >>> of one of the CPU's. Once this happens the server no longer responds to >>> requests. I've looked through the logs to try and see if anything stands >>> out but so far I've found nothing out of the ordinary. >>> >>> My current remedy is to log in and just kill the single processes that's >>> hung. Once that happens everything goes back to normal and I'm good for a >>> day or so. I'm currently the running following: >>> >>> solr-jetty-1.4.0+ds1-1ubuntu1 >>> >>> which is comprised of >>> >>> Solr 1.4.0 >>> Jetty 6.1.22 >>> on Unbuntu 10.10 >>> >>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just >>> looking for advice on how I should go about trouble shooting this problem. >>> >>> Chris >
Re: Debugging a Solr/Jetty Hung Process
OK... The fix I thought would fix it didn't fix it (which was to use the commitWithin feature). What I can gather from `ps` is that the thread has pages locked in memory. Currently I'm using native locking for Solr. Would switching to simple help alleviate this problem? Chris On Jun 4, 2011, at 2:48 PM, Chris Cowan wrote: > I found this thread that looks similar to what's happening on my system. I > think what happens is there are multiple commits happening at once from the > clients and it's causing the same issue. I'm going to use the commitWithin > argument to the updates to see if that fixes the problem. I will report back > with any findings. > > Chris > > On Jun 1, 2011, at 12:42 PM, Jonathan Rochkind wrote: > >> First guess (and it really is just a guess) would be Java garbage >> collection taking over. There are some JVM parameters you can use to >> tune the GC process, especially if the machine is multi-core, making >> sure GC happens in a seperate thread is helpful. >> >> But figuring out exactly what's going on requires confusing JVM >> debugging of which I am no expert at either. >> >> On 6/1/2011 3:04 PM, Chris Cowan wrote: >>> About once a day a Solr/Jetty process gets hung on my server consuming 100% >>> of one of the CPU's. Once this happens the server no longer responds to >>> requests. I've looked through the logs to try and see if anything stands >>> out but so far I've found nothing out of the ordinary. >>> >>> My current remedy is to log in and just kill the single processes that's >>> hung. Once that happens everything goes back to normal and I'm good for a >>> day or so. I'm currently the running following: >>> >>> solr-jetty-1.4.0+ds1-1ubuntu1 >>> >>> which is comprised of >>> >>> Solr 1.4.0 >>> Jetty 6.1.22 >>> on Unbuntu 10.10 >>> >>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just >>> looking for advice on how I should go about trouble shooting this problem. >>> >>> Chris >
