I have 'softAutoCommit' at 1 second and 'hardAutoCommit' at 30 seconds. On Mon, Jun 24, 2013 at 1:54 PM, Jason Hellman < jhell...@innoventsolutions.com> wrote:
> Vinay, > > What autoCommit settings do you have for your indexing process? > > Jason > > On Jun 24, 2013, at 1:28 PM, Vinay Pothnis <poth...@gmail.com> wrote: > > > Here is the ulimit -a output: > > > > core file size (blocks, -c) 0 data seg size > (kbytes, > > -d) unlimited scheduling priority (-e) 0 file size > > (blocks, -f) unlimited pending signals > > (-i) 179963 max locked memory (kbytes, -l) 64 max memory size > > (kbytes, -m) unlimited open files (-n) > > 32769 pipe size (512 bytes, -p) 8 POSIX message queues > > (bytes, > > -q) 819200 real-time priority (-r) 0 stack size > > (kbytes, -s) 10240 cpu time (seconds, -t) unlimited > max > > user processes (-u) 140000 virtual memory > (kbytes, > > -v) unlimited file locks (-x) unlimited > > > > On Mon, Jun 24, 2013 at 12:47 PM, Yago Riveiro <yago.rive...@gmail.com > >wrote: > > > >> Hi, > >> > >> I have the same issue too, and the deploy is quasi exact like than mine, > >> > http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067862 > >> > >> With some concurrence and batches of 10 solr apparently have some > deadlock > >> distributing updates > >> > >> Can you dump the configuration of the ulimit on your servers?, some > people > >> had the same issues because they are reach the ulimit maximum defined > for > >> descriptor and process. > >> > >> -- > >> Yago Riveiro > >> Sent with Sparrow (http://www.sparrowmailapp.com/?sig) > >> > >> > >> On Monday, June 24, 2013 at 7:49 PM, Vinay Pothnis wrote: > >> > >>> Hello All, > >>> > >>> I have the following set up of solr cloud. > >>> > >>> * solr version 4.3.1 > >>> * 3 node solr cloud + replciation factor 2 > >>> * 3 zoo keepers > >>> * load balancer in front of the 3 solr nodes > >>> > >>> I am seeing this strange behavior when I am indexing a large number of > >>> documents (10 mil). When I have more than 3-5 threads sending documents > >> (in > >>> batch of 20) to solr, sometimes solr goes into a hung state. After this > >> all > >>> the update requests get timed out. What we see via AppDynamics (a > >>> performance monitoring tool) is that there are a number of threads that > >> are > >>> stalled. The stack trace for one of the threads is shown below. > >>> > >>> The cluster has to be restarted to recover from this. When I reduce the > >>> concurrency to 1, 2, 3 threads, then the indexing goes through > smoothly. > >>> Any pointers as to what could be wrong here? > >>> > >>> We send the updates to one of the nodes in the solr cloud through a > load > >>> balancer. > >>> > >>> Thanks > >>> Vinay > >>> > >>> Thread Name:qtp2141131052-78 > >>> ID:78 > >>> Time:Fri Jun 21 23:20:22 GMT 2013 > >>> State:WAITING > >>> Priority:5 > >>> > >>> sun.misc.Unsafe.park(Native Method) > >>> java.util.concurrent.locks. > >>> LockSupport.park(LockSupport.java:186) > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > >>> > >> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > >>> java.util.concurrent.Semaphore.acquire(Semaphore.java:317) > >>> > >> > org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61) > >>> > >> > org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418) > >>> > >> > org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368) > >>> > >> > org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300) > >>> > >> > org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:96) > >>> > >> > org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:462) > >>> > >> > org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1178) > >>> > >> > org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179) > >>> > >> > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83) > >>> > >> > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > >>> org.apache.solr.core.SolrCore.execute(SolrCore.java:1820) > >>> > >> > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:656) > >>> > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359) > >>> > >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155) > >>> > >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1423) > >>> > >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:450) > >>> > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) > >>> > >> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564) > >>> > >> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) > >>> > >> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1083) > >>> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379) > >>> > >> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) > >>> > >> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017) > >>> > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) > >>> > >> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:258) > >>> > >> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) > >>> > >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > >>> org.eclipse.jetty.server.Server.handle(Server.java:445) > >>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260) > >>> > >> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225) > >>> > >> > org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) > >>> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:596) > >>> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:527) > >>> java.lang.Thread.run(Thread.java:722 > >>> > >>> > >> > >> > >> > >
