Hi, I have the same issue too, and the deploy is quasi exact like than mine, http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067862
With some concurrence and batches of 10 solr apparently have some deadlock distributing updates Can you dump the configuration of the ulimit on your servers?, some people had the same issues because they are reach the ulimit maximum defined for descriptor and process. -- Yago Riveiro Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Monday, June 24, 2013 at 7:49 PM, Vinay Pothnis wrote: > Hello All, > > I have the following set up of solr cloud. > > * solr version 4.3.1 > * 3 node solr cloud + replciation factor 2 > * 3 zoo keepers > * load balancer in front of the 3 solr nodes > > I am seeing this strange behavior when I am indexing a large number of > documents (10 mil). When I have more than 3-5 threads sending documents (in > batch of 20) to solr, sometimes solr goes into a hung state. After this all > the update requests get timed out. What we see via AppDynamics (a > performance monitoring tool) is that there are a number of threads that are > stalled. The stack trace for one of the threads is shown below. > > The cluster has to be restarted to recover from this. When I reduce the > concurrency to 1, 2, 3 threads, then the indexing goes through smoothly. > Any pointers as to what could be wrong here? > > We send the updates to one of the nodes in the solr cloud through a load > balancer. > > Thanks > Vinay > > Thread Name:qtp2141131052-78 > ID:78 > Time:Fri Jun 21 23:20:22 GMT 2013 > State:WAITING > Priority:5 > > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks. > LockSupport.park(LockSupport.java:186) > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > java.util.concurrent.Semaphore.acquire(Semaphore.java:317) > org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61) > org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418) > org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368) > org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300) > org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:96) > org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:462) > org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1178) > org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179) > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83) > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > org.apache.solr.core.SolrCore.execute(SolrCore.java:1820) > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:656) > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359) > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155) > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1423) > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:450) > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564) > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1083) > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379) > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017) > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:258) > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > org.eclipse.jetty.server.Server.handle(Server.java:445) > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260) > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225) > org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:596) > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:527) > java.lang.Thread.run(Thread.java:722 > >
