Hi
Thanks for the reply.
I get the information as following:
----------------------------------------------------
[solr@fkapp1 ~]$ ps mp 13359 -o THREAD,tid
USER %CPU PRI SCNT WCHAN USER SYSTEM TID
solr 217 - - - - - -
solr 0.0 21 - 184466 - - 13359
solr 0.0 19 - - - - 13360
solr 0.0 23 - 184466 - - 13361
.....
solr 99.9 14 - - - - 1210
solr 99.9 14 - - - - 1223
solr 99.9 14 - - - - 1227
solr 99.9 14 - - - - 1228
----------------------------------------------------------
Definitely, the suspicious threads are : 1210 1223 1227 1228, hexadecimal
values are 0x4ba,0x4c7,0x4cb,0x4cc,
And then get the thread info about the threads by the jstack tools.
----------------------------------------------------------
jstack -l 13359 > dump.stack
-----------------------------------------------------------
Finally, I find the stack info about the above threads:
I am not clear about the information, what does it mean?
Is there anything abnormal with the SolrDispatchFilter?
-----------------------------------------------------------
"http-bio-8081-exec-820" daemon prio=10 tid=0x00002aaac0d02800 nid=0x4cc
runnable [0x0000000043c87000]
java.lang.Thread.State: RUNNABLE
at java.util.WeakHashMap.put(WeakHashMap.java:405)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
- locked <0x0000000640604558> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x000000064061a360> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
"http-bio-8081-exec-802" daemon prio=10 tid=0x00002aaac0a03000 nid=0x4ba
runnable [0x0000000047dc8000]
java.lang.Thread.State: RUNNABLE
at java.util.WeakHashMap.get(WeakHashMap.java:355)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
- locked <0x0000000640605288> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x0000000640605308> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
......
-----------------------------------------------------------
2014/1/16 Otis Gospodnetic <otis.gospodne...@gmail.com>
> I wonder if this would be a good addition to Solr Admin functionality?
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Wed, Jan 15, 2014 at 6:29 AM, Mikhail Khludnev <
> mkhlud...@griddynamics.com> wrote:
>
> > Hello,
> >
> > Invoke top for particular process displaying threads enabled.
> > Find the hottest thread PID.
> > invoke jstack for this process, find the suspicious thread by "..
> > nid=0x[PID in hex]"
> > ...
> > PROFIT!
> >
> >
> > On Wed, Jan 15, 2014 at 1:38 PM, YouPeng Yang <yypvsxf19870...@gmail.com
> > >wrote:
> >
> > > Hi
> > > I find that the cpu ratio is very high when the tomcat contained solr
> > > 4.6 sleep.
> > > The pid 13359 shows that my sleeping solr web container take high cpu
> > > ratio
> > >
> > > Any insights?
> > >
> > >
> > > [solr@fkapp1 ~]$ top -d -1 -u solr
> > > top - 17:30:15 up 302 days, 7:10, 5 users, load average: 4.54, 4.52,
> > > 4.47
> > > Tasks: 418 total, 1 running, 412 sleeping, 0 stopped, 5 zombie
> > > Cpu(s): 19.1%us, 0.1%sy, 0.0%ni, 80.8%id, 0.0%wa, 0.0%hi, 0.0%si,
> > > 0.0%st
> > > Mem: 32955380k total, 28288212k used, 4667168k free, 503148k
> buffers
> > > Swap: 37257200k total, 87064k used, 37170136k free, 10861500k cached
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> > > COMMAND
> > >
> > > 13359 solr 21 0 11.4g 6.7g 12m S 400.5 21.4 491:30.85 java
> > >
> > > 3678 solr 15 0 13020 1380 828 R 0.0 0.0 0:19.16
> > > top
> > >
> > > 3694 solr 15 0 66092 1556 1228 S 0.0 0.0 0:00.01
> > > bash
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mkhlud...@griddynamics.com>
> >
>
