Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

Daniel de Oliveira Mantovani Fri, 01 Nov 2019 05:16:24 -0700

Hello Shalin,

It's not an OS bug,
Linux samba 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux


I'm sending the full thread dump log:
2019-11-01 11:05:13
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):

"Attach Listener" #259 daemon prio=9 os_prio=0 tid=0x00007fcb78001000
nid=0x2659 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"qtp1571967156-258" #258 prio=5 os_prio=0 tid=0x00007fcabc321000 nid=0x25c7
waiting on condition [0x00007fcb356ca000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005a626cdd8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-257" #257 prio=5 os_prio=0 tid=0x00007fcb24001000 nid=0x25bc
waiting on condition [0x00007fcb35b95000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f00242c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-256" #256 prio=5 os_prio=0 tid=0x00007fca88003000 nid=0x2594
runnable [0x00007fcb37441000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a34218> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a34200> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0931118> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-254" #254 prio=5 os_prio=0 tid=0x00007fca88005800 nid=0x2570
runnable [0x00007fcb3580f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a2fdd8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a2fdc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0243070> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-252" #252 prio=5 os_prio=0 tid=0x00007fca90008000 nid=0x2441
runnable [0x00007fc59c444000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a30180> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a30168> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0243220> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-250" #250 prio=5 os_prio=0 tid=0x00007fca90001000 nid=0x243d
runnable [0x00007fcb3570b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a313c8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a313b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da640> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-249" #249 prio=5 os_prio=0 tid=0x00007fca94001000 nid=0x2435
runnable [0x00007fcb373bf000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a30528> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a30510> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895ec0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-247" #247 prio=5 os_prio=0 tid=0x00007fcb18006000 nid=0x2428
runnable [0x00007fcb372bb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31770> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31758> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da6d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-245" #245 prio=5 os_prio=0 tid=0x00007fcb08004800 nid=0x2417
runnable [0x00007fcb36060000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31b18> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31b00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da5b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-243" #243 prio=5 os_prio=0 tid=0x00007fcae001f000 nid=0x240c
runnable [0x00007fcb35607000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31ec0> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31ea8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f06da490> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-241" #241 prio=5 os_prio=0 tid=0x00007fcb14001000 nid=0x23ff
runnable [0x00007fcb3727a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aabe8> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aabd0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242a88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-239" #239 prio=5 os_prio=0 tid=0x00007fcb0c001000 nid=0x23f1
runnable [0x00007fcb35648000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c4028> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4010> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895cc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-237" #237 prio=5 os_prio=0 tid=0x00007fca80001000 nid=0x23de
runnable [0x00007fc59ca8d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a34098> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a34080> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0930ed8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-235" #235 prio=5 os_prio=0 tid=0x00007fc588026000 nid=0x23d3
runnable [0x00007fcb35e99000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aac60> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aac48> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f02429f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-233" #233 prio=5 os_prio=0 tid=0x00007fcacc021000 nid=0x23c6
runnable [0x00007fcb35585000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aacc0> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aaca8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242ba8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-231" #231 prio=5 os_prio=0 tid=0x00007fc554004800 nid=0x23b9
runnable [0x00007fcb35891000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08aade0> (a sun.nio.ch.Util$3)
- locked <0x00000004f08aadc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242ad0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-228" #228 prio=5 os_prio=0 tid=0x00007fcacc015800 nid=0x23a4
runnable [0x00007fcb357ce000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a340f8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a340e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0930f68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-226" #226 prio=5 os_prio=0 tid=0x00007fca98005000 nid=0x2394
runnable [0x00007fcb35440000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c40e8> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c40d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895ba8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-224" #224 prio=5 os_prio=0 tid=0x00007fca60016800 nid=0x2388
runnable [0x00007fcb37482000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a34158> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a34140> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0930ff8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-222" #222 prio=5 os_prio=0 tid=0x00007fcb2c003000 nid=0x2378
runnable [0x00007fcb355c6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c4148> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4130> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895b18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-220" #220 prio=5 os_prio=0 tid=0x00007fca78003000 nid=0x236a
runnable [0x00007fcb374c3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a341b8> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a341a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0931088> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-218" #218 prio=5 os_prio=0 tid=0x00007fcb28002000 nid=0x235f
runnable [0x00007fcb360a1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c41a8> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4190> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895a88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-216" #216 prio=5 os_prio=0 tid=0x00007fcaec002000 nid=0x2353
waiting on condition [0x00007fc59c84c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000405560e38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-214" #214 prio=5 os_prio=0 tid=0x00007fc58c001800 nid=0x2341
waiting on condition [0x00007fcb37400000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f00242c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-202" #202 prio=5 os_prio=0 tid=0x00007fca88001000 nid=0x22f3
runnable [0x00007fcb358d2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a31020> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a31008> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895d10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-200" #200 prio=5 os_prio=0 tid=0x00007fcaa8002800 nid=0x22e7
runnable [0x00007fcb3574c000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a308d0> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a308b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895e30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-198" #198 prio=5 os_prio=0 tid=0x00007fca90007800 nid=0x22d7
runnable [0x00007fcb35689000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0a30c78> (a sun.nio.ch.Util$3)
- locked <0x00000004f0a30c60> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895da0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Scheduler-842741472" #89 prio=5 os_prio=0 tid=0x00007fca60007000
nid=0x1c62 waiting on condition [0x00007fc59cace000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0941af0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"SolrRrdBackendFactory-11-thread-2" #87 prio=5 os_prio=0
tid=0x00007fcac0001000 nid=0x1c60 waiting on condition [0x00007fcb35bd6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0996978> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"searcherExecutor-15-thread-1" #83 prio=5 os_prio=0 tid=0x00007fcab86b5800
nid=0x18a7 waiting on condition [0x00007fcb34e21000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f45eb9f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"searcherExecutor-14-thread-1" #84 prio=5 os_prio=0 tid=0x00007fcabc9c1000
nid=0x18a6 waiting on condition [0x00007fcb34de0000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f45eb710> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-78-acceptor-3@6942c593-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #78 prio=5 os_prio=0
tid=0x00007fca68001000 nid=0x1888 runnable [0x00007fcb35481000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #77 prio=5 os_prio=0
tid=0x00007fca74001000 nid=0x1886 waiting for monitor entry
[0x00007fcb354c2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #76 prio=5 os_prio=0
tid=0x00007fca70001000 nid=0x1885 waiting for monitor entry
[0x00007fcb35503000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
@362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}" #75 prio=5 os_prio=0
tid=0x00007fca7c001000 nid=0x1880 waiting for monitor entry
[0x00007fcb35544000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000004f08aab70> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-66" #66 prio=5 os_prio=0 tid=0x00007fca98001000 nid=0x186f
runnable [0x00007fcb3578d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f09c4088> (a sun.nio.ch.Util$3)
- locked <0x00000004f09c4070> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0895c38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"DestroyJavaVM" #61 prio=5 os_prio=0 tid=0x00007fcbd800e800 nid=0x175c
waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Thread-13" #58 prio=5 os_prio=0 tid=0x00007fcbd8f3f800 nid=0x1866 in
Object.wait() [0x00007fcb35b54000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850)
- locked <0x00000004f086a0f8> (a java.lang.Object)

   Locked ownable synchronizers:
- None

"MetricsHistoryHandler-12-thread-1" #55 prio=5 os_prio=0
tid=0x00007fcbd8ed3000 nid=0x185e waiting on condition [0x00007fcb35c17000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08abf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"SolrRrdBackendFactory-11-thread-1" #54 prio=5 os_prio=0
tid=0x00007fcbd8ed2000 nid=0x185d waiting on condition [0x00007fcb35c58000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0996978> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
#52 prio=5 os_prio=0 tid=0x00007fcbd83d8000 nid=0x185b waiting on condition
[0x00007fcb35eda000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08ac8e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
#51 daemon prio=5 os_prio=0 tid=0x00007fcbd83d6800 nid=0x185a runnable
[0x00007fcb35f1b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f09c7b38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
at
org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)
at
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018"
#47 daemon prio=5 os_prio=0 tid=0x00007fcbd83d5800 nid=0x1859 runnable
[0x00007fcb35f5c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0a2ef20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"zkConnectionManagerCallback-13-thread-1" #50 prio=5 os_prio=0
tid=0x00007fcad8001000 nid=0x1858 waiting on condition [0x00007fcb35f9d000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08ac430> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"main-EventThread" #49 daemon prio=5 os_prio=0 tid=0x00007fcbd8941800
nid=0x1857 waiting on condition [0x00007fcb35fde000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f0997bd8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

   Locked ownable synchronizers:
- None

"main-SendThread(localhost:2181)" #48 daemon prio=5 os_prio=0
tid=0x00007fcbd8948000 nid=0x1856 runnable [0x00007fcb3601f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f0997908> (a sun.nio.ch.Util$3)
- locked <0x00000004f09978f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f0242f08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)

   Locked ownable synchronizers:
- None

"zkConnectionManagerCallback-6-thread-1" #44 prio=5 os_prio=0
tid=0x00007fcaf4001000 nid=0x17a1 waiting on condition [0x00007fcb362e2000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08a9178> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"main-EventThread" #43 daemon prio=5 os_prio=0 tid=0x00007fcbd8db8800
nid=0x17a0 waiting on condition [0x00007fcb36323000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f08a95c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

   Locked ownable synchronizers:
- None

"main-SendThread(samba:2181)" #42 daemon prio=5 os_prio=0
tid=0x00007fcbd8db7000 nid=0x179f runnable [0x00007fcb36364000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000004f08a9898> (a sun.nio.ch.Util$3)
- locked <0x00000004f08a9880> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004f03ea3a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)

   Locked ownable synchronizers:
- None

"Connection evictor" #41 daemon prio=5 os_prio=0 tid=0x00007fcbd8daf800
nid=0x179e waiting on condition [0x00007fcb36c79000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Connection evictor" #40 daemon prio=5 os_prio=0 tid=0x00007fcbd8dae000
nid=0x179d waiting on condition [0x00007fcb36c38000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Connection evictor" #39 daemon prio=5 os_prio=0 tid=0x00007fcbd8da5000
nid=0x179c waiting on condition [0x00007fcb36cba000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Scheduler-1456339771" #35 prio=5 os_prio=0 tid=0x00007fcbd8ae3800
nid=0x178e waiting on condition [0x00007fcb36efb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000004f02dc358> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"ShutdownMonitor" #24 daemon prio=5 os_prio=0 tid=0x00007fcbd86e4800
nid=0x1783 runnable [0x00007fcb37704000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
org.eclipse.jetty.server.ShutdownMonitor$ShutdownMonitorRunnable.run(ShutdownMonitor.java:335)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"Service Thread" #21 daemon prio=9 os_prio=0 tid=0x00007fcbd8153800
nid=0x177b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread14" #20 daemon prio=9 os_prio=0 tid=0x00007fcbd8150800
nid=0x177a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread13" #19 daemon prio=9 os_prio=0 tid=0x00007fcbd814e000
nid=0x1779 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread12" #18 daemon prio=9 os_prio=0 tid=0x00007fcbd814c800
nid=0x1778 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread11" #17 daemon prio=9 os_prio=0 tid=0x00007fcbd814a000
nid=0x1777 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C1 CompilerThread10" #16 daemon prio=9 os_prio=0 tid=0x00007fcbd8148000
nid=0x1776 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread9" #15 daemon prio=9 os_prio=0 tid=0x00007fcbd8146800
nid=0x1775 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread8" #14 daemon prio=9 os_prio=0 tid=0x00007fcbd8144000
nid=0x1774 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread7" #13 daemon prio=9 os_prio=0 tid=0x00007fcbd8141800
nid=0x1773 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread6" #12 daemon prio=9 os_prio=0 tid=0x00007fcbd813f800
nid=0x1772 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread5" #11 daemon prio=9 os_prio=0 tid=0x00007fcbd813d800
nid=0x1771 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread4" #10 daemon prio=9 os_prio=0 tid=0x00007fcbd813b000
nid=0x1770 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007fcbd8131000
nid=0x176f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fcbd812f000
nid=0x176e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fcbd812c800
nid=0x176d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fcbd812a800
nid=0x176c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fcbd8128800
nid=0x176b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007fcbd8127000 nid=0x176a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fcbd80f6800 nid=0x1769 in
Object.wait() [0x00007fcbb807d000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000004f03573a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

   Locked ownable synchronizers:
- None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fcbd80f2000
nid=0x1768 in Object.wait() [0x00007fcbdc072000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000004f03574a8> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
- None

"VM Thread" os_prio=0 tid=0x00007fcbd80e8800 nid=0x1766 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8015000
nid=0x175d runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8021800
nid=0x175e runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8023000
nid=0x175f runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fcbd8024800
nid=0x1760 runnable

"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fcbd806e800
nid=0x1765 runnable

"Gang worker#0 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd8067800
nid=0x1761 runnable

"Gang worker#1 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd8069000
nid=0x1762 runnable

"Gang worker#2 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd806b000
nid=0x1763 runnable

"Gang worker#3 (Parallel CMS Threads)" os_prio=0 tid=0x00007fcbd806c800
nid=0x1764 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fcbd8156000 nid=0x177c
waiting on condition

JNI global references: 1002



Thank you

On Fri, Nov 1, 2019 at 8:26 AM Shalin Shekhar Mangar <shalinman...@gmail.com>
wrote:

> Perhaps this bug in the kernel?
> https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64
>
> Can you check if you are running an affected kernel?
>
> On Fri, Nov 1, 2019 at 7:09 AM Daniel de Oliveira Mantovani <
> daniel.oliveira.mantov...@gmail.com> wrote:
>
> > I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.
> >
> > $ java -version
> > java version "1.8.0_231"
> > Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode
> >
> > $ sudo strace -f -c -p 5972
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> >  71.02   33.329124         662     50335     13317 futex
> >  21.43   10.055086      335170        30        28 restart_syscall
> >   7.49    3.512846       14760       238           epoll_wait
> >   0.07    0.030526        2544        12           poll
> >   0.00    0.001571           5       322           write
> >   0.00    0.000439           4       117           times
> >   0.00    0.000355           2       183         4 read
> >   0.00    0.000236           3        72           mprotect
> >   0.00    0.000161           1       118           epoll_ctl
> >   0.00    0.000126           0       352           stat
> >   0.00    0.000089          11         8           getdents
> >   0.00    0.000077          10         8           sendto
> >   0.00    0.000069          69         1           statfs
> >   0.00    0.000043          22         2           writev
> >   0.00    0.000029           4         7           openat
> >   0.00    0.000021           4         6           fstat
> >   0.00    0.000018           1        22           lstat
> >   0.00    0.000018           9         2           dup
> >   0.00    0.000010           1         9           close
> >   0.00    0.000010           3         4           fcntl
> >   0.00    0.000005           3         2           access
> >   0.00    0.000002           2         1           mmap
> >   0.00    0.000002           0        26           sched_yield
> >   0.00    0.000001           0         8           recvfrom
> >   0.00    0.000001           1         2           readlink
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00   46.930865                 51887     13349 total
> >
> > $ sudo strace -f -p 5972
> > strace: Process 5972 attached with 86 threads
> > [pid  7524] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  7523] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  7511] epoll_wait(142,  <unfinished ...>
> > [pid  7471] epoll_wait(160,  <unfinished ...>
> > [pid  7266] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  7267] epoll_wait(151,  <unfinished ...>
> > [pid  7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  7265] epoll_wait(157,  <unfinished ...>
> > [pid  6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6281] epoll_wait(154,  <unfinished ...>
> > [pid  6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6277] accept(136,  <unfinished ...>
> > [pid  6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6271] epoll_wait(208,  <unfinished ...>
> > [pid  6270] epoll_wait(205,  <unfinished ...>
> > [pid  6262] epoll_wait(199,  <unfinished ...>
> > [pid  6261] epoll_wait(196,  <unfinished ...>
> > [pid  6258] epoll_wait(193,  <unfinished ...>
> > [pid  6269] epoll_wait(202,  <unfinished ...>
> > [pid  6257] epoll_wait(190,  <unfinished ...>
> > [pid  6256] epoll_wait(187,  <unfinished ...>
> > [pid  6255] epoll_wait(184,  <unfinished ...>
> > [pid  6253] epoll_wait(178,  <unfinished ...>
> > [pid  6250] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6252] epoll_wait(175,  <unfinished ...>
> > [pid  6251] epoll_wait(172,  <unfinished ...>
> > [pid  6254] epoll_wait(181,  <unfinished ...>
> > [pid  6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6238] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6237] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6234] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6233] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6230] epoll_wait(134,  <unfinished ...>
> > [pid  6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
> > ...>
> > [pid  6047] epoll_wait(130,  <unfinished ...>
> > [pid  6046] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid  6045] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> >
> > JVM args:
> >
> >
> >    -
> >
> -DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr.solr.home=/opt/solr/example/cloud/node1/solr-Duser.timezone=UTC-DzkClientTimeout=15000-DzkHost=localhost:2181-XX:+CMSParallelRemarkEnabled-XX:+CMSScavengeBeforeRemark-XX:+ParallelRefProcEnabled-XX:+PrintGCApplicationStoppedTime-XX:+PrintGCDateStamps-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintHeapAtGC-XX:+PrintTenuringDistribution-XX:+UseCMSInitiatingOccupancyOnly-XX:+UseConcMarkSweepGC-XX:+UseGCLogFileRotation-XX:-OmitStackTraceInFastThrow-XX:CMSInitiatingOccupancyFraction=50-XX:CMSMaxAbortablePrecleanTime=6000-XX:ConcGCThreads=4-XX:GCLogFileSize=20M-XX:MaxTenuringThreshold=8-XX:NewRatio=3-XX:NumberOfGCLogFiles=9-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
> >    8983
> >
> >
> /opt/solr/example/cloud/node1/solr/../logs-XX:ParallelGCThreads=4-XX:PretenureSizeThreshold=64m-XX:SurvivorRatio=4-XX:TargetSurvivorRatio=90-Xloggc:/opt/solr/example/cloud/node1/solr/../logs/solr_gc.log-Xms512m-Xmx15g-Xss256k-verbose:gc
> >
> > Java Thread Dumb:
> > name
> > cpuTime / userTime
> > process reaper (135)
> > java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> > (SynchronousQueue.java:460)
> > • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> > (SynchronousQueue.java:362)
> > • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1073)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 17.2219ms
> > 0.0000ms
> > qtp1571967156-129 (129)
> > 36.0159ms
> > 0.0000ms
> > process reaper (117)
> > java.util.concurrent.SynchronousQueue$TransferStack@70b8631c
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > • java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> > (SynchronousQueue.java:460)
> > • java.util.concurrent.SynchronousQueue$TransferStack.transfer
> > (SynchronousQueue.java:362)
> > • java.util.concurrent.SynchronousQueue.poll (SynchronousQueue.java:941)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1073)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 19.2751ms
> > 0.0000ms
> > qtp1571967156-116 (116)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.5766ms
> > 0.0000ms
> > qtp1571967156-115 (115)
> > • sun.management.ThreadImpl.getThreadInfo1 (Native Method)
> > • sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:176)
> > • org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody
> > (ThreadDumpHandler.java:71)
> > • org.apache.solr.handler.RequestHandlerBase.handleRequest
> > (RequestHandlerBase.java:199)
> > • org.apache.solr.handler.admin.InfoHandler.handle (InfoHandler.java:91)
> > • org.apache.solr.handler.admin.InfoHandler.handleRequestBody
> > (InfoHandler.java:81)
> > • org.apache.solr.handler.RequestHandlerBase.handleRequest
> > (RequestHandlerBase.java:199)
> > • org.apache.solr.servlet.HttpSolrCall.handleAdmin
> (HttpSolrCall.java:734)
> > • org.apache.solr.servlet.HttpSolrCall.handleAdminRequest
> > (HttpSolrCall.java:715)
> > • org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:496)
> > • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> > (SolrDispatchFilter.java:377)
> > • org.apache.solr.servlet.SolrDispatchFilter.doFilter
> > (SolrDispatchFilter.java:323)
> > • org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
> > (ServletHandler.java:1634)
> > • org.eclipse.jetty.servlet.ServletHandler.doHandle
> > (ServletHandler.java:533)
> > • org.eclipse.jetty.server.handler.ScopedHandler.handle
> > (ScopedHandler.java:146)
> > • org.eclipse.jetty.security.SecurityHandler.handle
> > (SecurityHandler.java:548)
> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> > (HandlerWrapper.java:132)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> > (ScopedHandler.java:257)
> > • org.eclipse.jetty.server.session.SessionHandler.doHandle
> > (SessionHandler.java:1595)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
> > (ScopedHandler.java:255)
> > • org.eclipse.jetty.server.handler.ContextHandler.doHandle
> > (ContextHandler.java:1317)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> > (ScopedHandler.java:203)
> > • org.eclipse.jetty.servlet.ServletHandler.doScope
> > (ServletHandler.java:473)
> > • org.eclipse.jetty.server.session.SessionHandler.doScope
> > (SessionHandler.java:1564)
> > • org.eclipse.jetty.server.handler.ScopedHandler.nextScope
> > (ScopedHandler.java:201)
> > • org.eclipse.jetty.server.handler.ContextHandler.doScope
> > (ContextHandler.java:1219)
> > • org.eclipse.jetty.server.handler.ScopedHandler.handle
> > (ScopedHandler.java:144)
> > • org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
> > (ContextHandlerCollection.java:219)
> > • org.eclipse.jetty.server.handler.HandlerCollection.handle
> > (HandlerCollection.java:126)
> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> > (HandlerWrapper.java:132)
> > • org.eclipse.jetty.rewrite.handler.RewriteHandler.handle
> > (RewriteHandler.java:335)
> > • org.eclipse.jetty.server.handler.HandlerWrapper.handle
> > (HandlerWrapper.java:132)
> > • org.eclipse.jetty.server.Server.handle (Server.java:531)
> > • org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
> > • org.eclipse.jetty.server.HttpConnection.onFillable
> > (HttpConnection.java:260)
> > • org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
> > (AbstractConnection.java:281)
> > • org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
> > • org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
> > (EatWhatYouKill.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:310)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 81.0773ms
> > 70.0000ms
> > qtp1571967156-114 (114)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 14.7759ms
> > 10.0000ms
> > qtp1571967156-113 (113)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.7541ms
> > 0.0000ms
> > qtp1571967156-112 (112)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 28.2561ms
> > 10.0000ms
> > qtp1571967156-109 (109)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 20.5324ms
> > 10.0000ms
> > qtp1571967156-107 (107)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 13.4180ms
> > 0.0000ms
> > qtp1571967156-90 (90)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 108.5163ms
> > 100.0000ms
> > Scheduler-842741472 (89)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@27022346
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1093)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.4794ms
> > 0.0000ms
> > qtp1571967156-88 (88)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 38.6417ms
> > 10.0000ms
> > SolrRrdBackendFactory-11-thread-2 (87)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1093)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 3.3288ms
> > 0.0000ms
> > searcherExecutor-15-thread-1 (83)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4004f6b2
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 8.9663ms
> > 0.0000ms
> > searcherExecutor-14-thread-1 (84)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7397cd2d
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 10.0677ms
> > 0.0000ms
> > qtp1571967156-79 (79)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 110.6471ms
> > 90.0000ms
> > qtp1571967156-78-acceptor-3@6942c593-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (78)
> > java.lang.Object@4539d80a
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:234)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 4.9023ms
> > 0.0000ms
> > qtp1571967156-77-acceptor-2@26493f4e-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (77)
> > java.lang.Object@4539d80a
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:234)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.0835ms
> > 0.0000ms
> > qtp1571967156-76-acceptor-1@292b8c3e-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (76)
> > • sun.nio.ch.ServerSocketChannelImpl.accept0 (Native Method)
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:422)
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:250)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.6738ms
> > 0.0000ms
> > qtp1571967156-75-acceptor-0@10c4f96f-ServerConnector
> > @362a019c{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} (75)
> > java.lang.Object@4539d80a
> > • sun.nio.ch.ServerSocketChannelImpl.accept
> > (ServerSocketChannelImpl.java:234)
> > • org.eclipse.jetty.server.ServerConnector.accept
> > (ServerConnector.java:369)
> > • org.eclipse.jetty.server.AbstractConnector$Acceptor.run
> > (AbstractConnector.java:639)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 4.9762ms
> > 0.0000ms
> > qtp1571967156-74 (74)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
> > (BlockingArrayQueue.java:392)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> > (QueuedThreadPool.java:653)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> > (QueuedThreadPool.java:48)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:717)
> > • java.lang.Thread.run (Thread.java:748)
> > 4.6754ms
> > 0.0000ms
> > qtp1571967156-73 (73)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
> > (BlockingArrayQueue.java:392)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> > (QueuedThreadPool.java:653)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> > (QueuedThreadPool.java:48)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:717)
> > • java.lang.Thread.run (Thread.java:748)
> > 6.0771ms
> > 0.0000ms
> > qtp1571967156-72 (72)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 28.1516ms
> > 10.0000ms
> > qtp1571967156-71 (71)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 31.7190ms
> > 10.0000ms
> > qtp1571967156-70 (70)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 22.8869ms
> > 0.0000ms
> > qtp1571967156-69 (69)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25ca8377
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.eclipse.jetty.util.BlockingArrayQueue.poll
> > (BlockingArrayQueue.java:392)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> > (QueuedThreadPool.java:653)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.access$800
> > (QueuedThreadPool.java:48)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:717)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.6719ms
> > 0.0000ms
> > qtp1571967156-68 (68)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 7.0843ms
> > 0.0000ms
> > qtp1571967156-67 (67)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 25.2077ms
> > 10.0000ms
> > qtp1571967156-66 (66)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 59.3490ms
> > 40.0000ms
> > qtp1571967156-65 (65)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 77.9195ms
> > 60.0000ms
> > qtp1571967156-64 (64)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 44.4205ms
> > 30.0000ms
> > qtp1571967156-63 (63)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:101)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select
> > (ManagedSelector.java:396)
> > • org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce
> > (ManagedSelector.java:333)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask
> > (EatWhatYouKill.java:357)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
> > (EatWhatYouKill.java:181)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
> > (EatWhatYouKill.java:168)
> > • org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
> > (EatWhatYouKill.java:126)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:366)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 129.4624ms
> > 90.0000ms
> > qtp1571967156-62 (62)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@421090bf
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2163)
> > •
> >
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait
> > (ReservedThreadExecutor.java:292)
> > • org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
> > (ReservedThreadExecutor.java:357)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> > (QueuedThreadPool.java:762)
> > • org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
> > (QueuedThreadPool.java:680)
> > • java.lang.Thread.run (Thread.java:748)
> > 51.2605ms
> > 30.0000ms
> > DestroyJavaVM (61)
> > •
> > 2002.8222ms
> > 1850.0000ms
> > Thread-13 (58)
> > java.lang.Object@f2f13c1
> > • java.lang.Object.wait (Native Method)
> > • java.lang.Object.wait (Object.java:502)
> > • org.apache.solr.core.CloserThread.run (CoreContainer.java:1850)
> > 0.7894ms
> > 0.0000ms
> > MetricsHistoryHandler-12-thread-1 (55)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@45d73a6
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1093)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 348.4029ms
> > 310.0000ms
> > SolrRrdBackendFactory-11-thread-1 (54)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7813101a
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:1088)
> > • java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
> > (ScheduledThreadPoolExecutor.java:809)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 5.7281ms
> > 0.0000ms
> >
> >
> OverseerAutoScalingTriggerThread-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> > (52)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75fd9961
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run
> > (OverseerTriggerThread.java:189)
> > • java.lang.Thread.run (Thread.java:748)
> > 58.1405ms
> > 50.0000ms
> >
> >
> OverseerCollectionConfigSetProcessor-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> > (51)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55c10b44
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> > (ZkDistributedQueue.java:463)
> > • org.apache.solr.cloud.OverseerTaskQueue.peekTopN
> > (OverseerTaskQueue.java:243)
> > • org.apache.solr.cloud.OverseerTaskProcessor.run
> > (OverseerTaskProcessor.java:233)
> > • java.lang.Thread.run (Thread.java:748)
> > 220.6397ms
> > 170.0000ms
> > OverseerStateUpdate-72060990031069185-192.168.1.40:8983_solr-n_0000001018
> > (47)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2287ab26
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> > •
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos
> > (AbstractQueuedSynchronizer.java:2078)
> > • org.apache.solr.cloud.ZkDistributedQueue.peekElements
> > (ZkDistributedQueue.java:463)
> > • org.apache.solr.cloud.Overseer$ClusterStateUpdater.run
> > (Overseer.java:207)
> > • java.lang.Thread.run (Thread.java:748)
> > 189.5649ms
> > 140.0000ms
> > zkConnectionManagerCallback-13-thread-1 (50)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da83c93
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.1550ms
> > 0.0000ms
> > main-EventThread (49)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@540792d6
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
> > 2.6375ms
> > 0.0000ms
> > main-SendThread(localhost:2181) (48)
> > • sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)
> > • sun.nio.ch.EPollArrayWrapper.poll (EPollArrayWrapper.java:269)
> > • sun.nio.ch.EPollSelectorImpl.doSelect (EPollSelectorImpl.java:93)
> > • sun.nio.ch.SelectorImpl.lockAndDoSelect (SelectorImpl.java:86)
> > • sun.nio.ch.SelectorImpl.select (SelectorImpl.java:97)
> > • org.apache.zookeeper.ClientCnxnSocketNIO.doTransport
> > (ClientCnxnSocketNIO.java:349)
> > • org.apache.zookeeper.ClientCnxn$SendThread.run (ClientCnxn.java:1144)
> > 52.9289ms
> > 10.0000ms
> > zkConnectionManagerCallback-6-thread-1 (44)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9c13839
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • java.util.concurrent.ThreadPoolExecutor.getTask
> > (ThreadPoolExecutor.java:1074)
> > • java.util.concurrent.ThreadPoolExecutor.runWorker
> > (ThreadPoolExecutor.java:1134)
> > • java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:624)
> > • java.lang.Thread.run (Thread.java:748)
> > 1.1128ms
> > 0.0000ms
> > main-EventThread (43)
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16806cbc
> > • sun.misc.Unsafe.park (Native Method)
> > • java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
> > •
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await
> > (AbstractQueuedSynchronizer.java:2039)
> > • java.util.concurrent.LinkedBlockingQueue.take
> > (LinkedBlockingQueue.java:442)
> > • org.apache.zookeeper.ClientCnxn$EventThread.run (ClientCnxn.java:502)
> >
> >
> > --
> >
> > --
> > Daniel Mantovani
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>


-- 

--
Daniel Mantovani

Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

Reply via email to