Perhaps this bug in the kernel?
https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64
Can you check if you are running an affected kernel?
On Fri, Nov 1, 2019 at 7:09 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantov...@gmail.com> wrote:
> I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.
>
> $ java -version
> java version "1.8.0_231"
> Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode
>
> $ sudo strace -f -c -p 5972
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 71.02 33.329124 662 50335 13317 futex
> 21.43 10.055086 335170 30 28 restart_syscall
> 7.49 3.512846 14760 238 epoll_wait
> 0.07 0.030526 2544 12 poll
> 0.00 0.001571 5 322 write
> 0.00 0.000439 4 117 times
> 0.00 0.000355 2 183 4 read
> 0.00 0.000236 3 72 mprotect
> 0.00 0.000161 1 118 epoll_ctl
> 0.00 0.000126 0 352 stat
> 0.00 0.000089 11 8 getdents
> 0.00 0.000077 10 8 sendto
> 0.00 0.000069 69 1 statfs
> 0.00 0.000043 22 2 writev
> 0.00 0.000029 4 7 openat
> 0.00 0.000021 4 6 fstat
> 0.00 0.000018 1 22 lstat
> 0.00 0.000018 9 2 dup
> 0.00 0.000010 1 9 close
> 0.00 0.000010 3 4 fcntl
> 0.00 0.000005 3 2 access
> 0.00 0.000002 2 1 mmap
> 0.00 0.000002 0 26 sched_yield
> 0.00 0.000001 0 8 recvfrom
> 0.00 0.000001 1 2 readlink
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 46.930865 51887 13349 total
>
> $ sudo strace -f -p 5972
> strace: Process 5972 attached with 86 threads
> [pid 7524] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 7523] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 7511] epoll_wait(142, <unfinished ...>
> [pid 7471] epoll_wait(160, <unfinished ...>
> [pid 7266] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 7267] epoll_wait(151, <unfinished ...>
> [pid 7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 7265] epoll_wait(157, <unfinished ...>
> [pid 6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6281] epoll_wait(154, <unfinished ...>
> [pid 6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6277] accept(136, <unfinished ...>
> [pid 6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6271] epoll_wait(208, <unfinished ...>
> [pid 6270] epoll_wait(205, <unfinished ...>
> [pid 6262] epoll_wait(199, <unfinished ...>
> [pid 6261] epoll_wait(196, <unfinished ...>
> [pid 6258] epoll_wait(193, <unfinished ...>
> [pid 6269] epoll_wait(202, <unfinished ...>
> [pid 6257] epoll_wait(190, <unfinished ...>
> [pid 6256] epoll_wait(187, <unfinished ...>
> [pid 6255] epoll_wait(184, <unfinished ...>
> [pid 6253] epoll_wait(178, <unfinished ...>
> [pid 6250] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 6252] epoll_wait(175, <unfinished ...>
> [pid 6251] epoll_wait(172, <unfinished ...>
> [pid 6254] epoll_wait(181, <unfinished ...>
> [pid 6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6238] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 6237] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6234] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 6233] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6230] epoll_wait(134, <unfinished ...>
> [pid 6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> ...>
> [pid 6047] epoll_wait(130, <unfinished ...>
> [pid 6046] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 6045] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
>
> JVM args:
>
>
> -
> -DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr.solr.home=/opt/solr/example/cloud/node1/solr-Duser.timezone=UTC-DzkClientTimeout=15000-DzkHost=localhost:2181-XX:+CMSParallelRemarkEnabled-XX:+CMSScavengeBeforeRemark-XX:+ParallelRefProcEnabled-XX:+PrintGCApplicationStoppedTime-XX:+PrintGCDateStamps-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintHeapAtGC-XX:+PrintTenuringDistribution-XX:+UseCMSInitiatingOccupancyOnly-XX:+UseConcMarkSweepGC-XX:+UseGCLogFileRotation-XX:-OmitStackTraceInFastThrow-XX:CMSInitiatingOccupancyFraction=50-XX:CMSMaxAbortablePrecleanTime=6000-XX:ConcGCThreads=4-XX:GCLogFileSize=20M-XX:MaxTenuringThreshold=8-XX:NewRatio=3-XX:NumberOfGCLogFiles=9-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
> 8983
>
>
> /opt/solr/example/cloud/node1/solr/../logs-XX:ParallelGCThreads=4-XX:PretenureSizeThreshold=64m-XX:SurvivorRatio=4-XX:TargetSurvivorRatio=90-Xloggc:/opt/solr/example/cloud/node1/solr/../logs/solr_gc.log-Xms512m-Xmx15g-Xss256k-verbose:gc
>
> Java Thread Dumb:
> name
> cpuTime / userTime
> process reaper (135)
> java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> (SynchronousQueue.java:460)
> • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> (SynchronousQueue.java:362)
> • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1073)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 17.2219ms
> 0.0000ms
> qtp1571967156-129 (129)
> 36.0159ms
> 0.0000ms
> process reaper (117)
> java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> (SynchronousQueue.java:460)
> • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> (SynchronousQueue.java:362)
> • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1073)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 19.2751ms
> 0.0000ms
> qtp1571967156-116 (116)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 1.5766ms
> 0.0000ms
> qtp1571967156-115 (115)
> • sun.management.ThreadImpl.getThreadInfo1 (Native Method)
> • sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:176)
> • org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody
> (ThreadDumpHandler.java:71)
> • org.apache.solr.handler.RequestHandlerBase.handleRequest
> (RequestHandlerBase.java:199)
> • org.apache.solr.handler.admin.InfoHandler.handle (InfoHandler.java:91)
> • org.apache.solr.handler.admin.InfoHandler.handleRequestBody
> (InfoHandler.java:81)
> • org.apache.solr.handler.RequestHandlerBase.handleRequest
> (RequestHandlerBase.java:199)
> • org.apache.solr.servlet.HttpSolrCall.handleAdmin (HttpSolrCall.java:734)
> • org.apache.solr.servlet.HttpSolrCall.handleAdminRequest
> (HttpSolrCall.java:715)
> • org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:496)
> • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> (SolrDispatchFilter.java:377)
> • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> (SolrDispatchFilter.java:323)
> • org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
> (ServletHandler.java:1634)
> • org.eclipse.jetty.servlet.ServletHandler.doHandle
> (ServletHandler.java:533)
> • org.eclipse.jetty.server.handler.ScopedHandler.handle
> (ScopedHandler.java:146)
> • org.eclipse.jetty.security.SecurityHandler.handle
> (SecurityHandler.java:548)
> • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> (HandlerWrapper.java:132)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> (ScopedHandler.java:257)
> • org.eclipse.jetty.server.session.SessionHandler.doHandle
> (SessionHandler.java:1595)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> (ScopedHandler.java:255)
> • org.eclipse.jetty.server.handler.ContextHandler.doHandle
> (ContextHandler.java:1317)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> (ScopedHandler.java:203)
> • org.eclipse.jetty.servlet.ServletHandler.doScope
> (ServletHandler.java:473)
> • org.eclipse.jetty.server.session.SessionHandler.doScope
> (SessionHandler.java:1564)
> • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> (ScopedHandler.java:201)
> • org.eclipse.jetty.server.handler.ContextHandler.doScope
> (ContextHandler.java:1219)
> • org.eclipse.jetty.server.handler.ScopedHandler.handle
> (ScopedHandler.java:144)
> • org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
> (ContextHandlerCollection.java:219)
> • org.eclipse.jetty.server.handler.HandlerCollection.handle
> (HandlerCollection.java:126)
> • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> (HandlerWrapper.java:132)
> • org.eclipse.jetty.rewrite.handler.RewriteHandler.handle
> (RewriteHandler.java:335)
> • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> (HandlerWrapper.java:132)
> • org.eclipse.jetty.server.Server.handle (Server.java:531)
> • org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
> • org.eclipse.jetty.server.HttpConnection.onFillable
> (HttpConnection.java:260)
> • org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
> (AbstractConnection.java:281)
> • org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
> • org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
> (EatWhatYouKill.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:310)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 81.0773ms
> 70.0000ms
> qtp1571967156-114 (114)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 14.7759ms
> 10.0000ms
> qtp1571967156-113 (113)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 1.7541ms
> 0.0000ms
> qtp1571967156-112 (112)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 28.2561ms
> 10.0000ms
> qtp1571967156-109 (109)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 20.5324ms
> 10.0000ms
> qtp1571967156-107 (107)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 13.4180ms
> 0.0000ms
> qtp1571967156-90 (90)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 108.5163ms
> 100.0000ms
> Scheduler-842741472 (89)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@27022346
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1093)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 5.4794ms
> 0.0000ms
> qtp1571967156-88 (88)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 38.6417ms
> 10.0000ms
> SolrRrdBackendFactory-11-thread-2 (87)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1093)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 3.3288ms
> 0.0000ms
> searcherExecutor-15-thread-1 (83)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4004f6b2
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 8.9663ms
> 0.0000ms
> searcherExecutor-14-thread-1 (84)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7397cd2d
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 10.0677ms
> 0.0000ms
> qtp1571967156-79 (79)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 110.6471ms
> 90.0000ms
> qtp1571967156-78-acceptor-3@6942c593-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (78)
> java.lang.Object@4539d80a
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:234)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 4.9023ms
> 0.0000ms
> qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (77)
> java.lang.Object@4539d80a
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:234)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 5.0835ms
> 0.0000ms
> qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (76)
> • sun.nio.ch.ServerSocketChannelImpl.accept0 (Native Method)
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:422)
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:250)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 5.6738ms
> 0.0000ms
> qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
> @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (75)
> java.lang.Object@4539d80a
> • sun.nio.ch.ServerSocketChannelImpl.accept
> (ServerSocketChannelImpl.java:234)
> • org.eclipse.jetty.server.ServerConnector.accept
> (ServerConnector.java:369)
> • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> (AbstractConnector.java:639)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 4.9762ms
> 0.0000ms
> qtp1571967156-74 (74)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.eclipse.jetty.util.BlockingArrayQueue.poll
> (BlockingArrayQueue.java:392)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> (QueuedThreadPool.java:653)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> (QueuedThreadPool.java:48)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:717)
> • java.lang.Thread.run (Thread.java:748)
> 4.6754ms
> 0.0000ms
> qtp1571967156-73 (73)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.eclipse.jetty.util.BlockingArrayQueue.poll
> (BlockingArrayQueue.java:392)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> (QueuedThreadPool.java:653)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> (QueuedThreadPool.java:48)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:717)
> • java.lang.Thread.run (Thread.java:748)
> 6.0771ms
> 0.0000ms
> qtp1571967156-72 (72)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 28.1516ms
> 10.0000ms
> qtp1571967156-71 (71)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 31.7190ms
> 10.0000ms
> qtp1571967156-70 (70)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 22.8869ms
> 0.0000ms
> qtp1571967156-69 (69)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.eclipse.jetty.util.BlockingArrayQueue.poll
> (BlockingArrayQueue.java:392)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> (QueuedThreadPool.java:653)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> (QueuedThreadPool.java:48)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:717)
> • java.lang.Thread.run (Thread.java:748)
> 5.6719ms
> 0.0000ms
> qtp1571967156-68 (68)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 7.0843ms
> 0.0000ms
> qtp1571967156-67 (67)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 25.2077ms
> 10.0000ms
> qtp1571967156-66 (66)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 59.3490ms
> 40.0000ms
> qtp1571967156-65 (65)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 77.9195ms
> 60.0000ms
> qtp1571967156-64 (64)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 44.4205ms
> 30.0000ms
> qtp1571967156-63 (63)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> (ManagedSelector.java:396)
> • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> (ManagedSelector.java:333)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> (EatWhatYouKill.java:357)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> (EatWhatYouKill.java:181)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> (EatWhatYouKill.java:168)
> • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> (EatWhatYouKill.java:126)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:366)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 129.4624ms
> 90.0000ms
> qtp1571967156-62 (62)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@421090bf
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2163)
> •
>
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait
> (ReservedThreadExecutor.java:292)
> • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> (ReservedThreadExecutor.java:357)
> • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:762)
> • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> (QueuedThreadPool.java:680)
> • java.lang.Thread.run (Thread.java:748)
> 51.2605ms
> 30.0000ms
> DestroyJavaVM (61)
> •
> 2002.8222ms
> 1850.0000ms
> Thread-13 (58)
> java.lang.Object@f2f13c1
> • java.lang.Object.wait (Native Method)
> • java.lang.Object.wait (Object.java:502)
> • org.apache.solr.core.CloserThread.run (CoreContainer.java:1850)
> 0.7894ms
> 0.0000ms
> MetricsHistoryHandler-12-thread-1 (55)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45d73a6
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1093)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 348.4029ms
> 310.0000ms
> SolrRrdBackendFactory-11-thread-1 (54)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:1088)
> • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> (ScheduledThreadPoolExecutor.java:809)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 5.7281ms
> 0.0000ms
>
> OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> (52)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75fd9961
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run
> (OverseerTriggerThread.java:189)
> • java.lang.Thread.run (Thread.java:748)
> 58.1405ms
> 50.0000ms
>
> OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> (51)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55c10b44
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> (ZkDistributedQueue.java:463)
> • org.apache.solr.cloud.OverseerTaskQueue.peekTopN
> (OverseerTaskQueue.java:243)
> • org.apache.solr.cloud.OverseerTaskProcessor.run
> (OverseerTaskProcessor.java:233)
> • java.lang.Thread.run (Thread.java:748)
> 220.6397ms
> 170.0000ms
> OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> (47)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2287ab26
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> •
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> (AbstractQueuedSynchronizer.java:2078)
> • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> (ZkDistributedQueue.java:463)
> • org.apache.solr.cloud.Overseer$ClusterStateUpdater.run
> (Overseer.java:207)
> • java.lang.Thread.run (Thread.java:748)
> 189.5649ms
> 140.0000ms
> zkConnectionManagerCallback-13-thread-1 (50)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da83c93
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 1.1550ms
> 0.0000ms
> main-EventThread (49)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@540792d6
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
> 2.6375ms
> 0.0000ms
> main-SendThread(localhost:2181) (48)
> • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> • org.apache.zookeeper.ClientCnxnSocketNIO.doTransport
> (ClientCnxnSocketNIO.java:349)
> • org.apache.zookeeper.ClientCnxn$SendThread.run (ClientCnxn.java:1144)
> 52.9289ms
> 10.0000ms
> zkConnectionManagerCallback-6-thread-1 (44)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9c13839
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • java.util.concurrent.ThreadPoolExecutor.getTask
> (ThreadPoolExecutor.java:1074)
> • java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1134)
> • java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> • java.lang.Thread.run (Thread.java:748)
> 1.1128ms
> 0.0000ms
> main-EventThread (43)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16806cbc
> • sun.misc.Unsafe.park (Native Method)
> • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> •
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> (AbstractQueuedSynchronizer.java:2039)
> • java.util.concurrent.LinkedBlockingQueue.take
> (LinkedBlockingQueue.java:442)
> • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
>
>
> --
>
> --
> Daniel Mantovani
>
--
Regards,
Shalin Shekhar Mangar.
