Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-26 Thread Vinay Pothnis
Thank you Erick! Will look at all these suggestions. -Vinay On Wed, Jun 26, 2013 at 6:37 AM, Erick Erickson wrote: > Right, unfortunately this is a gremlin lurking in the weeds, see: > http://wiki.apache.org/solr/DistributedSearch#Distributed_Deadlock > > There are a couple of ways to deal wit

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-26 Thread Erick Erickson
Right, unfortunately this is a gremlin lurking in the weeds, see: http://wiki.apache.org/solr/DistributedSearch#Distributed_Deadlock There are a couple of ways to deal with this: 1> go ahead and up the limit and re-compile, if you look at SolrCmdDistributor the semaphore is defined there. 2> http

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-25 Thread Vinay Pothnis
Jason and Scott, Thanks for the replies and pointers! Yes, I will consider the 'maxDocs' value as well. How do i monitor the transaction logs during the interval between commits? Thanks Vinay On Mon, Jun 24, 2013 at 8:48 PM, Jason Hellman < jhell...@innoventsolutions.com> wrote: > Scott, > > M

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Jason Hellman
Scott, My comment was meant to be a bit tongue-in-cheek, but my intent in the statement was to represent hard failure along the lines Vinay is seeing. We're talking about OutOfMemoryException conditions, total cluster paralysis requiring restart, or other similar and disastrous conditions. Wh

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Scott Lundgren
Jason, Regarding your statement "push you over the edge"- what does that mean? Does it mean "uncharted territory with unknown ramifications" or something more like specific, known symptoms? I ask because our use is similar to Vinay's in some respects, and we want to be able to push the capabilit

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Jason Hellman
Vinay, You may wish to pay attention to how many transaction logs are being created along the way to your hard autoCommit, which should truncate the open handles for those files. I might suggest setting a maxDocs value in parallel with your maxTime value (you can use both) to ensure the commit

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Vinay Pothnis
I have 'softAutoCommit' at 1 second and 'hardAutoCommit' at 30 seconds. On Mon, Jun 24, 2013 at 1:54 PM, Jason Hellman < jhell...@innoventsolutions.com> wrote: > Vinay, > > What autoCommit settings do you have for your indexing process? > > Jason > > On Jun 24, 2013, at 1:28 PM, Vinay Pothnis wr

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Jason Hellman
Vinay, What autoCommit settings do you have for your indexing process? Jason On Jun 24, 2013, at 1:28 PM, Vinay Pothnis wrote: > Here is the ulimit -a output: > > core file size (blocks, -c) 0 data seg size(kbytes, > -d) unlimited scheduling priority (-

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Vinay Pothnis
Here is the ulimit -a output: core file size (blocks, -c) 0 data seg size(kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 179963 max locked memory(kbytes, -l) 64 max memo

Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Yago Riveiro
Hi, I have the same issue too, and the deploy is quasi exact like than mine, http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067862 With some concurrence and batches of 10 solr apparently have some deadlock distributing updates Can you dump the configurat

[solr cloud] solr hangs when indexing large number of documents from multiple threads

2013-06-24 Thread Vinay Pothnis
Hello All, I have the following set up of solr cloud. * solr version 4.3.1 * 3 node solr cloud + replciation factor 2 * 3 zoo keepers * load balancer in front of the 3 solr nodes I am seeing this strange behavior when I am indexing a large number of documents (10 mil). When I have more than 3-5