Hi By the ways,after I restart the web container ,the ratio returns normal. So when does the sutuation come out?
Regards 2014/1/16 YouPeng Yang <yypvsxf19870...@gmail.com> > Hi > Thanks for the reply. > I get the information as following: > ---------------------------------------------------- > [solr@fkapp1 ~]$ ps mp 13359 -o THREAD,tid > USER %CPU PRI SCNT WCHAN USER SYSTEM TID > solr 217 - - - - - - > solr 0.0 21 - 184466 - - 13359 > solr 0.0 19 - - - - 13360 > solr 0.0 23 - 184466 - - 13361 > ..... > solr 99.9 14 - - - - 1210 > solr 99.9 14 - - - - 1223 > solr 99.9 14 - - - - 1227 > solr 99.9 14 - - - - 1228 > ---------------------------------------------------------- > Definitely, the suspicious threads are : 1210 1223 1227 1228, hexadecimal > values are 0x4ba,0x4c7,0x4cb,0x4cc, > > And then get the thread info about the threads by the jstack tools. > ---------------------------------------------------------- > jstack -l 13359 > dump.stack > > ----------------------------------------------------------- > > Finally, I find the stack info about the above threads: > > I am not clear about the information, what does it mean? > Is there anything abnormal with the SolrDispatchFilter? > > > > ----------------------------------------------------------- > "http-bio-8081-exec-820" daemon prio=10 tid=0x00002aaac0d02800 nid=0x4cc > runnable [0x0000000043c87000] > java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.put(WeakHashMap.java:405) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) > - locked <0x0000000640604558> (a > org.apache.tomcat.util.net.SocketWrapper) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - <0x000000064061a360> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "http-bio-8081-exec-802" daemon prio=10 tid=0x00002aaac0a03000 nid=0x4ba > runnable [0x0000000047dc8000] > java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:355) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) > - locked <0x0000000640605288> (a > org.apache.tomcat.util.net.SocketWrapper) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - <0x0000000640605308> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > > ...... > ----------------------------------------------------------- > > > > > 2014/1/16 Otis Gospodnetic <otis.gospodne...@gmail.com> > >> I wonder if this would be a good addition to Solr Admin functionality? >> >> Otis >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> On Wed, Jan 15, 2014 at 6:29 AM, Mikhail Khludnev < >> mkhlud...@griddynamics.com> wrote: >> >> > Hello, >> > >> > Invoke top for particular process displaying threads enabled. >> > Find the hottest thread PID. >> > invoke jstack for this process, find the suspicious thread by ".. >> > nid=0x[PID in hex]" >> > ... >> > PROFIT! >> > >> > >> > On Wed, Jan 15, 2014 at 1:38 PM, YouPeng Yang < >> yypvsxf19870...@gmail.com >> > >wrote: >> > >> > > Hi >> > > I find that the cpu ratio is very high when the tomcat contained >> solr >> > > 4.6 sleep. >> > > The pid 13359 shows that my sleeping solr web container take high >> cpu >> > > ratio >> > > >> > > Any insights? >> > > >> > > >> > > [solr@fkapp1 ~]$ top -d -1 -u solr >> > > top - 17:30:15 up 302 days, 7:10, 5 users, load average: 4.54, >> 4.52, >> > > 4.47 >> > > Tasks: 418 total, 1 running, 412 sleeping, 0 stopped, 5 zombie >> > > Cpu(s): 19.1%us, 0.1%sy, 0.0%ni, 80.8%id, 0.0%wa, 0.0%hi, 0.0%si, >> > > 0.0%st >> > > Mem: 32955380k total, 28288212k used, 4667168k free, 503148k >> buffers >> > > Swap: 37257200k total, 87064k used, 37170136k free, 10861500k >> cached >> > > >> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> > > COMMAND >> > > >> > > 13359 solr 21 0 11.4g 6.7g 12m S 400.5 21.4 491:30.85 java >> > > >> > > 3678 solr 15 0 13020 1380 828 R 0.0 0.0 0:19.16 >> > > top >> > > >> > > 3694 solr 15 0 66092 1556 1228 S 0.0 0.0 0:00.01 >> > > bash >> > > >> > >> > >> > >> > -- >> > Sincerely yours >> > Mikhail Khludnev >> > Principal Engineer, >> > Grid Dynamics >> > >> > <http://www.griddynamics.com> >> > <mkhlud...@griddynamics.com> >> > >> > >
