Re: Debugging update request

2013-10-18 Thread Erick Erickson
@Michael: Yep, that's the bit that's addressed by the two patches I referenced. If you can try this with 4.5 (or the soon to be done 4.5.1), the problem should go away. @Chris: I think you have a different issue. A very quick glance at your stack trace doesn't really show anything outstanding. T

Re: Debugging update request

2013-10-16 Thread Chris Geeringh
Here is my jstack output... Lots of blocked threads. http://pastebin.com/1ktjBYbf On 16 October 2013 10:28, michael.boom wrote: > I got the trace from jstack. > I found references to "semaphore" but not sure if this is what you meant. > Here's the trace: > http://pastebin.com/15QKAz7U > > > >

Re: Debugging update request

2013-10-16 Thread michael.boom
I got the trace from jstack. I found references to "semaphore" but not sure if this is what you meant. Here's the trace: http://pastebin.com/15QKAz7U -- View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095847.html Sent from the Solr - User mai

Re: Debugging update request

2013-10-16 Thread Chris Geeringh
I ran an import last night, and this morning my cloud wouldn't accept updates. I'm running the latest 4.6 snapshot. I was importing with latest solrj snapshot, and using java bin transport with CloudSolrServer. The cluster had indexed ~1.3 million docs before no further updates were accepted, quer

Re: Debugging update request

2013-10-16 Thread michael.boom
Thanks Erick! The version is 4.4.0. I'm posting 100k docs batches every 30-40 sec from each indexing client and sometimes two or more clients post in a very small timeframe. That's when i think the deadlock happens. I'll try to replicate the problem and check the thread dump. -- View this mes

Re: Debugging update request

2013-10-15 Thread Erick Erickson
What version of Solr? There's a known issue with distributed deadlock (since fixed) where under heavy load from multiple clients to multiple Solr nodes, you can get into a situation where it will stop accepting updates, but _will_ continue to serve queries. If this describes your situation, dump t