I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.
$ java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode
$ sudo strace -f -c -p 5972
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
71.02 33.329124 662 50335 13317 futex
21.43 10.055086 335170 30 28 restart_syscall
7.49 3.512846 14760 238 epoll_wait
0.07 0.030526 2544 12 poll
0.00 0.001571 5 322 write
0.00 0.000439 4 117 times
0.00 0.000355 2 183 4 read
0.00 0.000236 3 72 mprotect
0.00 0.000161 1 118 epoll_ctl
0.00 0.000126 0 352 stat
0.00 0.000089 11 8 getdents
0.00 0.000077 10 8 sendto
0.00 0.000069 69 1 statfs
0.00 0.000043 22 2 writev
0.00 0.000029 4 7 openat
0.00 0.000021 4 6 fstat
0.00 0.000018 1 22 lstat
0.00 0.000018 9 2 dup
0.00 0.000010 1 9 close
0.00 0.000010 3 4 fcntl
0.00 0.000005 3 2 access
0.00 0.000002 2 1 mmap
0.00 0.000002 0 26 sched_yield
0.00 0.000001 0 8 recvfrom
0.00 0.000001 1 2 readlink
------ ----------- ----------- --------- --------- ----------------
100.00 46.930865 51887 13349 total
$ sudo strace -f -p 5972
strace: Process 5972 attached with 86 threads
[pid 7524] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 7523] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 7511] epoll_wait(142, <unfinished ...>
[pid 7471] epoll_wait(160, <unfinished ...>
[pid 7266] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 7267] epoll_wait(151, <unfinished ...>
[pid 7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 7265] epoll_wait(157, <unfinished ...>
[pid 6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6281] epoll_wait(154, <unfinished ...>
[pid 6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6277] accept(136, <unfinished ...>
[pid 6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6271] epoll_wait(208, <unfinished ...>
[pid 6270] epoll_wait(205, <unfinished ...>
[pid 6262] epoll_wait(199, <unfinished ...>
[pid 6261] epoll_wait(196, <unfinished ...>
[pid 6258] epoll_wait(193, <unfinished ...>
[pid 6269] epoll_wait(202, <unfinished ...>
[pid 6257] epoll_wait(190, <unfinished ...>
[pid 6256] epoll_wait(187, <unfinished ...>
[pid 6255] epoll_wait(184, <unfinished ...>
[pid 6253] epoll_wait(178, <unfinished ...>
[pid 6250] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 6252] epoll_wait(175, <unfinished ...>
[pid 6251] epoll_wait(172, <unfinished ...>
[pid 6254] epoll_wait(181, <unfinished ...>
[pid 6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6238] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 6237] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6234] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 6233] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6230] epoll_wait(134, <unfinished ...>
[pid 6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 6047] epoll_wait(130, <unfinished ...>
[pid 6046] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 6045] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
JVM args:
-
-DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr.solr.home=/opt/solr/example/cloud/node1/solr-Duser.timezone=UTC-DzkClientTimeout=15000-DzkHost=localhost:2181-XX:+CMSParallelRemarkEnabled-XX:+CMSScavengeBeforeRemark-XX:+ParallelRefProcEnabled-XX:+PrintGCApplicationStoppedTime-XX:+PrintGCDateStamps-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintHeapAtGC-XX:+PrintTenuringDistribution-XX:+UseCMSInitiatingOccupancyOnly-XX:+UseConcMarkSweepGC-XX:+UseGCLogFileRotation-XX:-OmitStackTraceInFastThrow-XX:CMSInitiatingOccupancyFraction=50-XX:CMSMaxAbortablePrecleanTime=6000-XX:ConcGCThreads=4-XX:GCLogFileSize=20M-XX:MaxTenuringThreshold=8-XX:NewRatio=3-XX:NumberOfGCLogFiles=9-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
8983
/opt/solr/example/cloud/node1/solr/../logs-XX:ParallelGCThreads=4-XX:PretenureSizeThreshold=64m-XX:SurvivorRatio=4-XX:TargetSurvivorRatio=90-Xloggc:/opt/solr/example/cloud/node1/solr/../logs/solr_gc.log-Xms512m-Xmx15g-Xss256k-verbose:gc
Java Thread Dumb:
name
cpuTime / userTime
process reaper (135)
java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
• java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
(SynchronousQueue.java:460)
• java.util.concurrent.SynchronousQueue$TransferStack.transfer
(SynchronousQueue.java:362)
• java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1073)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
17.2219ms
0.0000ms
qtp1571967156-129 (129)
36.0159ms
0.0000ms
process reaper (117)
java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
• java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
(SynchronousQueue.java:460)
• java.util.concurrent.SynchronousQueue$TransferStack.transfer
(SynchronousQueue.java:362)
• java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1073)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
19.2751ms
0.0000ms
qtp1571967156-116 (116)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
1.5766ms
0.0000ms
qtp1571967156-115 (115)
• sun.management.ThreadImpl.getThreadInfo1 (Native Method)
• sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:176)
• org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody
(ThreadDumpHandler.java:71)
• org.apache.solr.handler.RequestHandlerBase.handleRequest
(RequestHandlerBase.java:199)
• org.apache.solr.handler.admin.InfoHandler.handle (InfoHandler.java:91)
• org.apache.solr.handler.admin.InfoHandler.handleRequestBody
(InfoHandler.java:81)
• org.apache.solr.handler.RequestHandlerBase.handleRequest
(RequestHandlerBase.java:199)
• org.apache.solr.servlet.HttpSolrCall.handleAdmin (HttpSolrCall.java:734)
• org.apache.solr.servlet.HttpSolrCall.handleAdminRequest
(HttpSolrCall.java:715)
• org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:496)
• org.apache.solr.servlet.SolrDispatchFilter.doFilter
(SolrDispatchFilter.java:377)
• org.apache.solr.servlet.SolrDispatchFilter.doFilter
(SolrDispatchFilter.java:323)
• org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
(ServletHandler.java:1634)
• org.eclipse.jetty.servlet.ServletHandler.doHandle
(ServletHandler.java:533)
• org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:146)
• org.eclipse.jetty.security.SecurityHandler.handle
(SecurityHandler.java:548)
• org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
• org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:257)
• org.eclipse.jetty.server.session.SessionHandler.doHandle
(SessionHandler.java:1595)
• org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
(ScopedHandler.java:255)
• org.eclipse.jetty.server.handler.ContextHandler.doHandle
(ContextHandler.java:1317)
• org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:203)
• org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473)
• org.eclipse.jetty.server.session.SessionHandler.doScope
(SessionHandler.java:1564)
• org.eclipse.jetty.server.handler.ScopedHandler.nextScope
(ScopedHandler.java:201)
• org.eclipse.jetty.server.handler.ContextHandler.doScope
(ContextHandler.java:1219)
• org.eclipse.jetty.server.handler.ScopedHandler.handle
(ScopedHandler.java:144)
• org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
(ContextHandlerCollection.java:219)
• org.eclipse.jetty.server.handler.HandlerCollection.handle
(HandlerCollection.java:126)
• org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
• org.eclipse.jetty.rewrite.handler.RewriteHandler.handle
(RewriteHandler.java:335)
• org.eclipse.jetty.server.handler.HandlerWrapper.handle
(HandlerWrapper.java:132)
• org.eclipse.jetty.server.Server.handle (Server.java:531)
• org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
• org.eclipse.jetty.server.HttpConnection.onFillable
(HttpConnection.java:260)
• org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
(AbstractConnection.java:281)
• org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
• org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
(EatWhatYouKill.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:310)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
81.0773ms
70.0000ms
qtp1571967156-114 (114)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
14.7759ms
10.0000ms
qtp1571967156-113 (113)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
1.7541ms
0.0000ms
qtp1571967156-112 (112)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
28.2561ms
10.0000ms
qtp1571967156-109 (109)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
20.5324ms
10.0000ms
qtp1571967156-107 (107)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
13.4180ms
0.0000ms
qtp1571967156-90 (90)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
108.5163ms
100.0000ms
Scheduler-842741472 (89)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@27022346
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1093)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
5.4794ms
0.0000ms
qtp1571967156-88 (88)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
38.6417ms
10.0000ms
SolrRrdBackendFactory-11-thread-2 (87)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1093)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
3.3288ms
0.0000ms
searcherExecutor-15-thread-1 (83)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4004f6b2
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
8.9663ms
0.0000ms
searcherExecutor-14-thread-1 (84)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7397cd2d
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
10.0677ms
0.0000ms
qtp1571967156-79 (79)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
110.6471ms
90.0000ms
qtp1571967156-78-acceptor-3@6942c593-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (78)
java.lang.Object@4539d80a
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:234)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
4.9023ms
0.0000ms
qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (77)
java.lang.Object@4539d80a
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:234)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
5.0835ms
0.0000ms
qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (76)
• sun.nio.ch.ServerSocketChannelImpl.accept0 (Native Method)
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:422)
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:250)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
5.6738ms
0.0000ms
qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (75)
java.lang.Object@4539d80a
• sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:234)
• org.eclipse.jetty.server.ServerConnector.accept (ServerConnector.java:369)
• org.eclipse.jetty.server.AbstractConnector$Acceptor.run
(AbstractConnector.java:639)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
4.9762ms
0.0000ms
qtp1571967156-74 (74)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.eclipse.jetty.util.BlockingArrayQueue.poll
(BlockingArrayQueue.java:392)
• org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
(QueuedThreadPool.java:653)
• org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
(QueuedThreadPool.java:48)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:717)
• java.lang.Thread.run (Thread.java:748)
4.6754ms
0.0000ms
qtp1571967156-73 (73)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.eclipse.jetty.util.BlockingArrayQueue.poll
(BlockingArrayQueue.java:392)
• org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
(QueuedThreadPool.java:653)
• org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
(QueuedThreadPool.java:48)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:717)
• java.lang.Thread.run (Thread.java:748)
6.0771ms
0.0000ms
qtp1571967156-72 (72)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
28.1516ms
10.0000ms
qtp1571967156-71 (71)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
31.7190ms
10.0000ms
qtp1571967156-70 (70)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
22.8869ms
0.0000ms
qtp1571967156-69 (69)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.eclipse.jetty.util.BlockingArrayQueue.poll
(BlockingArrayQueue.java:392)
• org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
(QueuedThreadPool.java:653)
• org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
(QueuedThreadPool.java:48)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:717)
• java.lang.Thread.run (Thread.java:748)
5.6719ms
0.0000ms
qtp1571967156-68 (68)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
7.0843ms
0.0000ms
qtp1571967156-67 (67)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
25.2077ms
10.0000ms
qtp1571967156-66 (66)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
59.3490ms
40.0000ms
qtp1571967156-65 (65)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
77.9195ms
60.0000ms
qtp1571967156-64 (64)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
44.4205ms
30.0000ms
qtp1571967156-63 (63)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
(ManagedSelector.java:396)
• org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
(ManagedSelector.java:333)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
(EatWhatYouKill.java:357)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
(EatWhatYouKill.java:181)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
(EatWhatYouKill.java:168)
• org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
(EatWhatYouKill.java:126)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:366)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
129.4624ms
90.0000ms
qtp1571967156-62 (62)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@421090bf
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2163)
•
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait
(ReservedThreadExecutor.java:292)
• org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
(ReservedThreadExecutor.java:357)
• org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
(QueuedThreadPool.java:762)
• org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
(QueuedThreadPool.java:680)
• java.lang.Thread.run (Thread.java:748)
51.2605ms
30.0000ms
DestroyJavaVM (61)
•
2002.8222ms
1850.0000ms
Thread-13 (58)
java.lang.Object@f2f13c1
• java.lang.Object.wait (Native Method)
• java.lang.Object.wait (Object.java:502)
• org.apache.solr.core.CloserThread.run (CoreContainer.java:1850)
0.7894ms
0.0000ms
MetricsHistoryHandler-12-thread-1 (55)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45d73a6
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1093)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
348.4029ms
310.0000ms
SolrRrdBackendFactory-11-thread-1 (54)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1088)
• java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
5.7281ms
0.0000ms
OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018
(52)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75fd9961
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run
(OverseerTriggerThread.java:189)
• java.lang.Thread.run (Thread.java:748)
58.1405ms
50.0000ms
OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018
(51)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55c10b44
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.apache.solr.cloud.ZkDistributedQueue.peekElements
(ZkDistributedQueue.java:463)
• org.apache.solr.cloud.OverseerTaskQueue.peekTopN
(OverseerTaskQueue.java:243)
• org.apache.solr.cloud.OverseerTaskProcessor.run
(OverseerTaskProcessor.java:233)
• java.lang.Thread.run (Thread.java:748)
220.6397ms
170.0000ms
OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018
(47)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2287ab26
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
(AbstractQueuedSynchronizer.java:2078)
• org.apache.solr.cloud.ZkDistributedQueue.peekElements
(ZkDistributedQueue.java:463)
• org.apache.solr.cloud.Overseer$ClusterStateUpdater.run (Overseer.java:207)
• java.lang.Thread.run (Thread.java:748)
189.5649ms
140.0000ms
zkConnectionManagerCallback-13-thread-1 (50)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da83c93
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
1.1550ms
0.0000ms
main-EventThread (49)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@540792d6
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
2.6375ms
0.0000ms
main-SendThread(localhost:2181) (48)
• sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
• sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
• sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
• sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
• sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
• org.apache.zookeeper.ClientCnxnSocketNIO.doTransport
(ClientCnxnSocketNIO.java:349)
• org.apache.zookeeper.ClientCnxn$SendThread.run (ClientCnxn.java:1144)
52.9289ms
10.0000ms
zkConnectionManagerCallback-6-thread-1 (44)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9c13839
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1074)
• java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1134)
• java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:624)
• java.lang.Thread.run (Thread.java:748)
1.1128ms
0.0000ms
main-EventThread (43)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16806cbc
• sun.misc.Unsafe.park (Native Method)
• java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
•
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
(AbstractQueuedSynchronizer.java:2039)
• java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
• org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
--
--
Daniel Mantovani
