Has anyone changed your tomcat settings? You're logging information at the INFO level, I wonder if you used to be logging at WARN.
I'd also take a look at your log directory to see if you've got a bazillion log files, and/or how big your log file is, has it been accumulating log messages forever? In other words, are you absolutely sure that your log file handling is correct? Take a look at how much free disk space you have at this point. Like I said above, this is a long shot but often one of the things that doesn't show up for a very long time on a production system. And why are you logging to console anyway? Has that changed from file-only? Or am I on a completely wrong tangent??? Best Erick On Tue, Aug 24, 2010 at 6:41 AM, AlexxelA <alexandre.boudrea...@canoe.ca>wrote: > > Thread dump > > Got like 240 thread like this : > > > "http-8080-Processor222" daemon prio=10 tid=0x00007fe36c010c00 nid=0x1e94 > waiting for monitor entry [0x000000004caa6000..0x000000004caa6d20] > java.lang.Thread.State: BLOCKED (on object monitor) > at java.util.logging.StreamHandler.publish(StreamHandler.java:174) > - waiting to lock <0x00007fe37e72b340> (a > java.util.logging.ConsoleHandler) > at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88) > at java.util.logging.Logger.log(Logger.java:472) > at java.util.logging.Logger.doLog(Logger.java:494) > at java.util.logging.Logger.log(Logger.java:517) > at java.util.logging.Logger.info(Logger.java:1036) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212) > at > > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874) > at > > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) > at > > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) > at > > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) > at > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) > at java.lang.Thread.run(Thread.java:619) > > > Here the running thread : > > "http-8080-Processor156" daemon prio=10 tid=0x0000000000df2000 nid=0x1e52 > runnable [0x0000000044521000..0x0000000044521c20] > java.lang.Thread.State: RUNNABLE > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) > - locked <0x00007fe37e3abcd8> (a java.io.BufferedOutputStream) > at java.io.PrintStream.write(PrintStream.java:430) > - locked <0x00007fe37e3abca0> (a java.io.PrintStream) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) > at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) > at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276) > at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) > - locked <0x00007fe37e72cd90> (a java.io.OutputStreamWriter) > at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) > at java.util.logging.StreamHandler.flush(StreamHandler.java:225) > - locked <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler) > at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89) > at java.util.logging.Logger.log(Logger.java:472) > at java.util.logging.Logger.doLog(Logger.java:494) > at java.util.logging.Logger.log(Logger.java:517) > at java.util.logging.Logger.info(Logger.java:1036) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212) > at > > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874) > at > > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) > at > > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) > at > > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) > at > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) > at java.lang.Thread.run(Thread.java:619) > > And the system thread > > "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 > tid=0x0000000000b1c800 nid=0x192 waiting on condition > [0x000000004183b000..0x000000004183bba0] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1548) > at java.lang.Thread.run(Thread.java:619) > > "pool-1-thread-1" prio=10 tid=0x00007fe374220400 nid=0x191 waiting for > monitor entry [0x0000000040bca000..0x0000000040bcaa20] > java.lang.Thread.State: BLOCKED (on object monitor) > at java.util.logging.StreamHandler.publish(StreamHandler.java:174) > - waiting to lock <0x00007fe37e72b340> (a > java.util.logging.ConsoleHandler) > at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88) > at java.util.logging.Logger.log(Logger.java:472) > at java.util.logging.Logger.doLog(Logger.java:494) > at java.util.logging.Logger.log(Logger.java:517) > at java.util.logging.Logger.info(Logger.java:1036) > at > org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1519) > at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > > "Low Memory Detector" daemon prio=10 tid=0x00007fe374005000 nid=0x18e > runnable [0x0000000000000000..0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "CompilerThread1" daemon prio=10 tid=0x00007fe374002800 nid=0x18d waiting > on > condition [0x0000000000000000..0x00000000403684d0] > java.lang.Thread.State: RUNNABLE > > "CompilerThread0" daemon prio=10 tid=0x00007fe374000c00 nid=0x18c waiting > on > condition [0x0000000000000000..0x0000000040267540] > java.lang.Thread.State: RUNNABLE > > "Signal Dispatcher" daemon prio=10 tid=0x00000000006bb000 nid=0x18b > runnable > [0x0000000000000000..0x00000000401677b0] > java.lang.Thread.State: RUNNABLE > > "Finalizer" daemon prio=10 tid=0x0000000000690c00 nid=0x18a in > Object.wait() > [0x000000004152d000..0x000000004152dba0] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00007fe37e3a21d0> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) > - locked <0x00007fe37e3a21d0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > > "Reference Handler" daemon prio=10 tid=0x000000000068f400 nid=0x189 in > Object.wait() [0x000000004142c000..0x000000004142ca20] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00007fe37e3a2208> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:485) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x00007fe37e3a2208> (a java.lang.ref.Reference$Lock) > > "main" prio=10 tid=0x0000000000622400 nid=0x16e runnable > [0x0000000000000000..0x00007fffcd94d9b0] > java.lang.Thread.State: RUNNABLE > > "VM Thread" prio=10 tid=0x000000000068a000 nid=0x188 runnable > > "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000062cc00 nid=0x178 > runnable > > "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000000062e000 nid=0x179 > runnable > > "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000000062f000 nid=0x17a > runnable > > "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000630400 nid=0x17b > runnable > > "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000000631800 nid=0x17c > runnable > > "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000000632c00 nid=0x17e > runnable > > "GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000000634000 nid=0x17f > runnable > > "GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000000635400 nid=0x180 > runnable > > "VM Periodic Task Thread" prio=10 tid=0x00007fe374007400 nid=0x18f waiting > on condition > > JNI global references: 1151 > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-jam-after-all-my-jvm-thread-pool-hang-in-blocked-state-tp1303361p1311364.html > Sent from the Solr - User mailing list archive at Nabble.com. >
