Vinay, You may wish to pay attention to how many transaction logs are being created along the way to your hard autoCommit, which should truncate the open handles for those files. I might suggest setting a maxDocs value in parallel with your maxTime value (you can use both) to ensure the commit occurs at either breakpoint. 30 seconds is plenty of time for 5 parallel processes of 20 document submissions to push you over the edge.
Jason On Jun 24, 2013, at 2:21 PM, Vinay Pothnis <poth...@gmail.com> wrote: > I have 'softAutoCommit' at 1 second and 'hardAutoCommit' at 30 seconds. > > On Mon, Jun 24, 2013 at 1:54 PM, Jason Hellman < > jhell...@innoventsolutions.com> wrote: > >> Vinay, >> >> What autoCommit settings do you have for your indexing process? >> >> Jason >> >> On Jun 24, 2013, at 1:28 PM, Vinay Pothnis <poth...@gmail.com> wrote: >> >>> Here is the ulimit -a output: >>> >>> core file size (blocks, -c) 0 data seg size >> (kbytes, >>> -d) unlimited scheduling priority (-e) 0 file size >>> (blocks, -f) unlimited pending signals >>> (-i) 179963 max locked memory (kbytes, -l) 64 max memory size >>> (kbytes, -m) unlimited open files (-n) >>> 32769 pipe size (512 bytes, -p) 8 POSIX message queues >>> (bytes, >>> -q) 819200 real-time priority (-r) 0 stack size >>> (kbytes, -s) 10240 cpu time (seconds, -t) unlimited >> max >>> user processes (-u) 140000 virtual memory >> (kbytes, >>> -v) unlimited file locks (-x) unlimited >>> >>> On Mon, Jun 24, 2013 at 12:47 PM, Yago Riveiro <yago.rive...@gmail.com >>> wrote: >>> >>>> Hi, >>>> >>>> I have the same issue too, and the deploy is quasi exact like than mine, >>>> >> http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067862 >>>> >>>> With some concurrence and batches of 10 solr apparently have some >> deadlock >>>> distributing updates >>>> >>>> Can you dump the configuration of the ulimit on your servers?, some >> people >>>> had the same issues because they are reach the ulimit maximum defined >> for >>>> descriptor and process. >>>> >>>> -- >>>> Yago Riveiro >>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig) >>>> >>>> >>>> On Monday, June 24, 2013 at 7:49 PM, Vinay Pothnis wrote: >>>> >>>>> Hello All, >>>>> >>>>> I have the following set up of solr cloud. >>>>> >>>>> * solr version 4.3.1 >>>>> * 3 node solr cloud + replciation factor 2 >>>>> * 3 zoo keepers >>>>> * load balancer in front of the 3 solr nodes >>>>> >>>>> I am seeing this strange behavior when I am indexing a large number of >>>>> documents (10 mil). When I have more than 3-5 threads sending documents >>>> (in >>>>> batch of 20) to solr, sometimes solr goes into a hung state. After this >>>> all >>>>> the update requests get timed out. What we see via AppDynamics (a >>>>> performance monitoring tool) is that there are a number of threads that >>>> are >>>>> stalled. The stack trace for one of the threads is shown below. >>>>> >>>>> The cluster has to be restarted to recover from this. When I reduce the >>>>> concurrency to 1, 2, 3 threads, then the indexing goes through >> smoothly. >>>>> Any pointers as to what could be wrong here? >>>>> >>>>> We send the updates to one of the nodes in the solr cloud through a >> load >>>>> balancer. >>>>> >>>>> Thanks >>>>> Vinay >>>>> >>>>> Thread Name:qtp2141131052-78 >>>>> ID:78 >>>>> Time:Fri Jun 21 23:20:22 GMT 2013 >>>>> State:WAITING >>>>> Priority:5 >>>>> >>>>> sun.misc.Unsafe.park(Native Method) >>>>> java.util.concurrent.locks. >>>>> LockSupport.park(LockSupport.java:186) >>>>> >>>> >> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) >>>>> >>>> >> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) >>>>> >>>> >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) >>>>> java.util.concurrent.Semaphore.acquire(Semaphore.java:317) >>>>> >>>> >> org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61) >>>>> >>>> >> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418) >>>>> >>>> >> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368) >>>>> >>>> >> org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300) >>>>> >>>> >> org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:96) >>>>> >>>> >> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:462) >>>>> >>>> >> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1178) >>>>> >>>> >> org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179) >>>>> >>>> >> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83) >>>>> >>>> >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) >>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:1820) >>>>> >>>> >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:656) >>>>> >>>> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359) >>>>> >>>> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155) >>>>> >>>> >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1423) >>>>> >>>> >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:450) >>>>> >>>> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) >>>>> >>>> >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564) >>>>> >>>> >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) >>>>> >>>> >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1083) >>>>> >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379) >>>>> >>>> >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) >>>>> >>>> >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017) >>>>> >>>> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) >>>>> >>>> >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:258) >>>>> >>>> >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) >>>>> >>>> >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) >>>>> org.eclipse.jetty.server.Server.handle(Server.java:445) >>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260) >>>>> >>>> >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225) >>>>> >>>> >> org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) >>>>> >>>> >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:596) >>>>> >>>> >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:527) >>>>> java.lang.Thread.run(Thread.java:722 >>>>> >>>>> >>>> >>>> >>>> >> >>
