Hello,
I can't say anything from this thread dump, but it's really suspicious
stacks:
java.lang.Thread.State: RUNNABLE
at java.util.WeakHashMap.get(
WeakHashMap.java:355)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
java.lang.Thread.State: RUNNABLE
at java.util.WeakHashMap.put(
WeakHashMap.java:405)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
Nevertheless, it disproves your initial statement about sleeping solr - it
handled requests, might not be regular ones. Now you can find in log which
requests those threads handled at that time.
How many cores you have in this instance? Are you sure that everything was
fine with the heap?
On Thu, Jan 16, 2014 at 11:36 AM, YouPeng Yang <yypvsxf19870...@gmail.com>wrote:
> Hi
> Thanks for the reply.
> I get the information as following:
> ----------------------------------------------------
> [solr@fkapp1 ~]$ ps mp 13359 -o THREAD,tid
> USER %CPU PRI SCNT WCHAN USER SYSTEM TID
> solr 217 - - - - - -
> solr 0.0 21 - 184466 - - 13359
> solr 0.0 19 - - - - 13360
> solr 0.0 23 - 184466 - - 13361
> .....
> solr 99.9 14 - - - - 1210
> solr 99.9 14 - - - - 1223
> solr 99.9 14 - - - - 1227
> solr 99.9 14 - - - - 1228
> ----------------------------------------------------------
> Definitely, the suspicious threads are : 1210 1223 1227 1228, hexadecimal
> values are 0x4ba,0x4c7,0x4cb,0x4cc,
>
> And then get the thread info about the threads by the jstack tools.
> ----------------------------------------------------------
> jstack -l 13359 > dump.stack
>
> -----------------------------------------------------------
>
> Finally, I find the stack info about the above threads:
>
> I am not clear about the information, what does it mean?
> Is there anything abnormal with the SolrDispatchFilter?
>
>
>
> -----------------------------------------------------------
> "http-bio-8081-exec-820" daemon prio=10 tid=0x00002aaac0d02800 nid=0x4cc
> runnable [0x0000000043c87000]
> java.lang.Thread.State: RUNNABLE
> at java.util.WeakHashMap.put(WeakHashMap.java:405)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
> at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
> at
>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at
>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> - locked <0x0000000640604558> (a
> org.apache.tomcat.util.net.SocketWrapper)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - <0x000000064061a360> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> "http-bio-8081-exec-802" daemon prio=10 tid=0x00002aaac0a03000 nid=0x4ba
> runnable [0x0000000047dc8000]
> java.lang.Thread.State: RUNNABLE
> at java.util.WeakHashMap.get(WeakHashMap.java:355)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
> at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
> at
>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at
>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> - locked <0x0000000640605288> (a
> org.apache.tomcat.util.net.SocketWrapper)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - <0x0000000640605308> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> ......
> -----------------------------------------------------------
>
>
>
>
> 2014/1/16 Otis Gospodnetic <otis.gospodne...@gmail.com>
>
> > I wonder if this would be a good addition to Solr Admin functionality?
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Wed, Jan 15, 2014 at 6:29 AM, Mikhail Khludnev <
> > mkhlud...@griddynamics.com> wrote:
> >
> > > Hello,
> > >
> > > Invoke top for particular process displaying threads enabled.
> > > Find the hottest thread PID.
> > > invoke jstack for this process, find the suspicious thread by "..
> > > nid=0x[PID in hex]"
> > > ...
> > > PROFIT!
> > >
> > >
> > > On Wed, Jan 15, 2014 at 1:38 PM, YouPeng Yang <
> yypvsxf19870...@gmail.com
> > > >wrote:
> > >
> > > > Hi
> > > > I find that the cpu ratio is very high when the tomcat contained
> solr
> > > > 4.6 sleep.
> > > > The pid 13359 shows that my sleeping solr web container take high
> cpu
> > > > ratio
> > > >
> > > > Any insights?
> > > >
> > > >
> > > > [solr@fkapp1 ~]$ top -d -1 -u solr
> > > > top - 17:30:15 up 302 days, 7:10, 5 users, load average: 4.54,
> 4.52,
> > > > 4.47
> > > > Tasks: 418 total, 1 running, 412 sleeping, 0 stopped, 5 zombie
> > > > Cpu(s): 19.1%us, 0.1%sy, 0.0%ni, 80.8%id, 0.0%wa, 0.0%hi,
> 0.0%si,
> > > > 0.0%st
> > > > Mem: 32955380k total, 28288212k used, 4667168k free, 503148k
> > buffers
> > > > Swap: 37257200k total, 87064k used, 37170136k free, 10861500k
> cached
> > > >
> > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> > > > COMMAND
> > > >
> > > > 13359 solr 21 0 11.4g 6.7g 12m S 400.5 21.4 491:30.85 java
> > > >
> > > > 3678 solr 15 0 13020 1380 828 R 0.0 0.0 0:19.16
> > > > top
> > > >
> > > > 3694 solr 15 0 66092 1556 1228 S 0.0 0.0 0:00.01
> > > > bash
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > > Principal Engineer,
> > > Grid Dynamics
> > >
> > > <http://www.griddynamics.com>
> > > <mkhlud...@griddynamics.com>
> > >
> >
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mkhlud...@griddynamics.com>
