high cpu threads (solr 7.5) - EPollArrayWrapper.epollWait
Version: solr cloud 7.5 OS: CentOS 7 JDK: Oracle JDK 1.8.0_191 We are noticing high CPU utilization on below threads. Looks like a known issue with. (https://github.com/netty/netty/issues/327) But not sure if this has been addressed in any of the 1.8 releases. Please help. "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 nid=0x4996 runnable [0x7f51fc6d8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00064cded430> (a sun.nio.ch.Util$3) - locked <0x00064cded418> (a java.util.Collections$UnmodifiableSet) - locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748)
high cpu threads (solr 7.5)
We are noticing high CPU utilization on below threads. Looks like a known issue with. (https://github.com/netty/netty/issues/327) But not sure if this has been addressed in any of the 1.8 releases. Can anyone help with this? Version: solr cloud 7.5 OS: CentOS 7 JDK: Oracle JDK 1.8.0_191 "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 nid=0x4996 runnable [0x7f51fc6d8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00064cded430> (a sun.nio.ch.Util$3) - locked <0x00064cded418> (a java.util.Collections$UnmodifiableSet) - locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748)
Re: high cpu threads (solr 7.5)
Thank you. We are planning to upgrade the JDK 11. Is solr 7.5 fully compatible with openjdk 11. On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson wrote: > It hasn’t been addressed by any Java 8 releases that I know of. > > See: https://issues.apache.org/jira/browse/SOLR-13349 > > The work-around in Solr is trivial, see the patch so it’d be simple to > patch/compile on your own. > > It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of > which have been released yet. > > Or move to Java 9 or later. > > > On Apr 3, 2019, at 4:39 PM, Hari Nakka wrote: > > > > We are noticing high CPU utilization on below threads. Looks like a > known > > issue with. (https://github.com/netty/netty/issues/327) > > > > But not sure if this has been addressed in any of the 1.8 releases. > > > > Can anyone help with this? > > > > > > Version: solr cloud 7.5 > > > > OS: CentOS 7 > > > > JDK: Oracle JDK 1.8.0_191 > > > > > > > > > > > > "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 > > nid=0x4996 runnable [0x7f51fc6d8000] > > > > java.lang.Thread.State: RUNNABLE > > > >at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > >at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > >at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > >at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > >- locked <0x00064cded430> (a sun.nio.ch.Util$3) > > > >- locked <0x00064cded418> (a > > java.util.Collections$UnmodifiableSet) > > > >- locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) > > > >at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > >at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > > > >at > > org.eclipse.jetty.io > .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) > > > >at > > org.eclipse.jetty.io > .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > > > >at > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > > > >at > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > > > >at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) > > > >at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) > > > >at java.lang.Thread.run(Thread.java:748) > >
Solr7.5 and zookeeper 3.4.14
We have solr 7.5 cloud setup with external zookeeper ensemble 3.4.11 (as it is the recommended version bundled with the distribution) I read new zookeeper versions are backward compatible. Can we upgrade it to 3.4.14. Any issue reported ?
Re: high cpu threads (solr 7.5)
Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu utilization randomly. Attached the full threaddump (tdump.out) and lwp utilization (high-cpu.out) there were more than 30 threads (high-cpu-dump.out)taking high cpu. these are different threads. i couldn't find much looking at them. Can you take a look and see if there is any known condition that could be causing this. On Sat, Apr 6, 2019 at 12:23 PM Erick Erickson wrote: > Here’s what the current state of various Solr x Java versions: > https://wiki.apache.org/solr/SolrJavaVersions > > > On Apr 5, 2019, at 3:16 PM, Hari Nakka wrote: > > > > Thank you. We are planning to upgrade the JDK 11. > > Is solr 7.5 fully compatible with openjdk 11. > > > > > > On Thu, Apr 4, 2019 at 9:58 AM Erick Erickson > > wrote: > > > >> It hasn’t been addressed by any Java 8 releases that I know of. > >> > >> See: https://issues.apache.org/jira/browse/SOLR-13349 > >> > >> The work-around in Solr is trivial, see the patch so it’d be simple to > >> patch/compile on your own. > >> > >> It will be released in a Solr 7.7.2 and Solr 8.1 or later, neither of > >> which have been released yet. > >> > >> Or move to Java 9 or later. > >> > >>> On Apr 3, 2019, at 4:39 PM, Hari Nakka wrote: > >>> > >>> We are noticing high CPU utilization on below threads. Looks like a > >> known > >>> issue with. (https://github.com/netty/netty/issues/327) > >>> > >>> But not sure if this has been addressed in any of the 1.8 releases. > >>> > >>> Can anyone help with this? > >>> > >>> > >>> Version: solr cloud 7.5 > >>> > >>> OS: CentOS 7 > >>> > >>> JDK: Oracle JDK 1.8.0_191 > >>> > >>> > >>> > >>> > >>> > >>> "qtp574568002-3821728" #3821728 prio=5 os_prio=0 tid=0x7f4f20018000 > >>> nid=0x4996 runnable [0x7f51fc6d8000] > >>> > >>> java.lang.Thread.State: RUNNABLE > >>> > >>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > >>> > >>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > >>> > >>> at > >> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > >>> > >>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > >>> > >>> - locked <0x00064cded430> (a sun.nio.ch.Util$3) > >>> > >>> - locked <0x00064cded418> (a > >>> java.util.Collections$UnmodifiableSet) > >>> > >>> - locked <0x00064cdf6e38> (a sun.nio.ch.EPollSelectorImpl) > >>> > >>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > >>> > >>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > >>> > >>> at > >>> org.eclipse.jetty.io > >> .ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) > >>> > >>> at > >>> org.eclipse.jetty.io > >> .ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) > >>> > >>> at > >>> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) > >>> > >>> at java.lang.Thread.run(Thread.java:748) > >> > >> > >
Re: high cpu threads (solr 7.5)
I mean the light weight processes (lwp) which were taking high cpu. I pulled the actual threads taking high cpu. full thread dump: *tdump.out* linux lwps: *high-cpu.out* top high cpu lwps mapped to thread nid: *high-cpu-dump.out (included threads taking more than 50% virtual core cpu)* https://drive.google.com/drive/folders/1FsFb8l0u5ZV4qnO-tgqa91JPHqT0lCcR On Thu, Apr 11, 2019 at 1:03 PM Shawn Heisey wrote: > On 4/11/2019 2:21 AM, Hari Nakka wrote: > > Hi Erick, We upgraded JDK to 11. No improvement. Still seeing high cpu > > utilization randomly. > > Attached the full threaddump (tdump.out) and lwp utilization > (high-cpu.out) > > there were more than 30 threads (high-cpu-dump.out)taking high cpu. > > these are different threads. i couldn't find much looking at them. > > Can you take a look and see if there is any known condition that could > > be causing this. > > Maybe we need a prominent note on the "mailing lists" page about > attachments to the mailing list, saying something like the following: > > --- > Attachments rarely make it to the list. They are stripped by the > mailing list software. If you have files to share with the list, you'll > need to find another way to send them. File sharing websites usually > work well. > --- > > We didn't get any attachments from you. > > You said you included something called "lwp utilization" ... the only > lwp I have ever heard of is a perl module for HTTP, which would have > nothing at all to do with high CPU usage in Solr. > > Thread dumps are usually not helpful in diagnosing high CPU usage. They > contain zero information about which threads are consuming the most CPU. > > Thanks, > Shawn >
