I apologize for the delay. The replication stalling out doesn't happen daily. I will paste the thread dump below to try to help. This is on a server that is currently locked on replication for a few hours. Any more information please let me know. There are no errors in the logs either so very strange.
Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23 Thread Count: current=73, peak=125, daemon=70 Full Thread Dump: 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms user time=23940.0000ms at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav a:182) at java.io.FilterInputStream.read(FilterInputStream.java:116) at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream .java:108) at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1 22) at org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java: 808) at org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764 ) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255) at org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav a:224) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri odic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch eduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 07) at java.lang.Thread.run(Thread.java:619) 'pool-16-thread-1' Id=237, WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-190' Id=232, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-189' Id=231, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-188' Id=230, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-187' Id=229, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-186' Id=228, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-182' Id=224, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-181' Id=223, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-180' Id=222, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-179' Id=221, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-178' Id=220, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-177' Id=219, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-175' Id=217, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-173' Id=215, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-171' Id=213, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-170' Id=212, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-169' Id=211, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-168' Id=210, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-167' Id=209, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-166' Id=208, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-165' Id=207, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-164' Id=206, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-161' Id=203, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-160' Id=202, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-159' Id=201, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-158' Id=200, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-157' Id=199, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-156' Id=198, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu time=1800610.0000ms user time=1743610.0000ms at sun.management.ThreadImpl.getThreadInfo0(Native Method) at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145) at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3 74) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application FilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh ain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher. java:630) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis patcher.java:436) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch er.java:374) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher .java:302) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java: 273) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application FilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh ain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja va:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja va:175) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128 ) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102 ) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java :109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java: 879) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H ttp11NioProtocol.java:719) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java: 2080) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 07) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-153' Id=195, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-152' Id=194, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-150' Id=192, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-149' Id=191, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-147' Id=189, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-146' Id=188, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-145' Id=187, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-144' Id=186, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-143' Id=185, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-140' Id=182, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-137' Id=179, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-136' Id=178, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-133' Id=175, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-132' Id=174, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-131' Id=173, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-128' Id=170, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-127' Id=169, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-126' Id=165, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-125' Id=164, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8443-exec-5' Id=152, WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@1 34ad83f, total cpu time=110.0000ms user time=80.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8443-exec-4' Id=151, WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@1 34ad83f, total cpu time=120.0000ms user time=80.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8443-exec-3' Id=150, WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@1 34ad83f, total cpu time=120.0000ms user time=80.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8443-exec-2' Id=149, WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@1 34ad83f, total cpu time=110.0000ms user time=80.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8443-exec-1' Id=148, WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@1 34ad83f, total cpu time=140.0000ms user time=110.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-107' Id=129, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-88' Id=110, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-65' Id=87, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'http-8080-exec-6' Id=28, TIMED_WAITING on lock=java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@5 bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN anos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 06) at java.lang.Thread.run(Thread.java:619) 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on lock=java.lang.ref.referencequeue$l...@16bb19fe, total cpu time=0.0000ms user time=0.0000ms at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu eueThread.run(MultiThreadedHttpConnectionManager.java:1122) 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu time=31410.0000ms user time=24650.0000ms at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473) at java.lang.Thread.run(Thread.java:619) 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms user time=250.0000ms at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198) at java.lang.Thread.run(Thread.java:619) 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu time=764410.0000ms user time=394670.0000ms at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473) at java.lang.Thread.run(Thread.java:619) 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=128740.0000ms user time=95550.0000ms at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198) at java.lang.Thread.run(Thread.java:619) 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16, TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont ainerBase.java:1579) at java.lang.Thread.run(Thread.java:619) 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu time=17960.0000ms user time=11850.0000ms at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe lector.java:305) 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu time=261720.0000ms user time=120120.0000ms at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe lector.java:305) 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user time=0.0000ms at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer SocketFactory.java:34) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport .java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:619) 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user time=0.0000ms at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport .java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:619) 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user time=0.0000ms at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport .java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:619) 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user time=0.0000ms 'Finalizer' Id=3, WAITING on lock=java.lang.ref.referencequeue$l...@55a77e5a, total cpu time=16460.0000ms user time=15430.0000ms at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) 'Reference Handler' Id=2, WAITING on lock=java.lang.ref.reference$l...@675872a4, total cpu time=4630.0000ms user time=3390.0000ms at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user time=1570.0000ms at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.catalina.core.StandardServer.await(StandardServer.java:389) at org.apache.catalina.startup.Catalina.await(Catalina.java:642) at org.apache.catalina.startup.Catalina.start(Catalina.java:602) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) -- Jeff Newburn Software Engineer, Zappos.com jnewb...@zappos.com - 702-943-7562 > From: Noble Paul നോബിള് नोब्ळ् <noble.p...@gmail.com> > Reply-To: <solr-user@lucene.apache.org> > Date: Tue, 24 Mar 2009 14:33:50 +0530 > To: <solr-user@lucene.apache.org> > Subject: Re: <no subject> > > We do not set an conn_timeoout,read_timeout for the httpclient in snappuller. > > I guess it should be set to some very high value say 1hr for > read-timeout and say 1 minute for conn_timeout and we can make it > configurable . > > --Noble > > On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar > <shalinman...@gmail.com> wrote: >> We should obviously get to the bottom of this. But I was thinking, should we >> have some sort of timeouts on the SnapPuller in the slave to avoid such >> scenarios? Locking out snap pulls forever is not a good idea. >> >> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley >> <yo...@lucidimagination.com>wrote: >> >>> So this is only one slave that hangs up and not the master? >>> Can you get thread dumps on both the master and the slave during a hang? >>> >>> >>> -Yonik >>> http://www.lucidimagination.com >>> >>> >>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <jnewb...@zappos.com> >>> wrote: >>>> We are having an intermittent problem with replication. We reindex >>> nightly >>>> which usually means there are 2 commits during replication then a final >>>> commit/optimize at the end. For some reason the replication will hang >>>> occasionally with the following screenshot. This is frustrating as it >>> will >>>> completely stall out any further replications. Additionally, it seems to >>>> only happen on reindex and it will strike 1 server randomly but not >>> always >>>> the same server. >>>> >>>> >>>> In case the screen shot doesn’t come through: >>>> >>>> Master http://10.66.209.38:8080/solr/zeta-main/replication >>>> Latest Index Version:1233423827699, Generation: 6237 >>>> Replicatable Index Version:0, Generation: 0 >>>> Poll Interval 00:05:00 >>>> Local Index Index Version: 1233423827684, Generation: 6222 >>>> Location: /opt/solr-data/zeta-main/index >>>> Size: 1.29 GB >>>> Times Replicated Since Startup: 3591 >>>> Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009 >>>> Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009 >>>> Config Files Replicated: [synonyms.txt] >>>> Times Config Files Replicated Since Startup: 4 >>>> Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009 >>>> Current Replication Status Start Time: Mon Mar 23 00:22:55 PDT 2009 >>>> Files Downloaded: 12 / 163 >>>> Downloaded: 4.12 MB / 1.41 GB [0.0%] >>>> Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%] >>>> Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163 >>>> bytes/s >>>> >>>> >>>> >>>> -- >>>> Jeff Newburn >>>> Software Engineer, Zappos.com >>>> jnewb...@zappos.com - 702-943-7562 >>>> >>> >> >> >> >> -- >> Regards, >> Shalin Shekhar Mangar. >> > > > > -- > --Noble Paul
