https://issues.apache.org/bugzilla/show_bug.cgi?id=50306
--- Comment #7 from TomLu <thoma...@tihlde.org> 2010-12-29 05:02:04 EST --- Created an attachment (id=26451) --> (https://issues.apache.org/bugzilla/attachment.cgi?id=26451) Patch for Tomcat's ThreadPoolExecutor WebSphere has hung/stuck thread detection logic that has been very useful to us, both in development and production. But since we are moving away from WebSphere, it would be nice to have this kind of feature in Tomcat as well. Some might consider this "stuck thread detection" unnessesary bloat, but we have found it useful on numerous occasions. Applications are never perfect, and often threads will run for too long (resulting in thread leakage and pool depletion): -External connections might hang (like URLConnection) [you can configure read/connect timeout, but not all applications has configured this] -Loops (can be detected by excessive CPU usage) -Loops which does not have excessive CPU usage (for example because it uses external slow IO in each loop iteration, or has Thread.sleep!! (bad coding, but it can happen)) -Deadlocks. -Threads waiting on other hung threads. I have created an experimental patch for Tomcat 7.0.5 that we have been using in development for a short time now. The hung thread monitoring does the following: Monitors active (managed) threads, and issues a warning for each thread that has been active for more than the configured threshold (seconds). The message (log.warn) contains the thread name, stack trace, total time active, and total number of suspected hung threads. (In our environment, we use Cron to scan system.out, and send email if a hung thread is detected.) A timer thread runs at a configurable interval and checks and reports hung threads, and threads previously reported to be hung, but have now finished. It detects hung threads in the HTTP/AJP thread pools. It does not monitor all JVM threads, but since many user threads are spawned from managed "webrequest" threads, the HTTP/AJP threads often waits for user threads to finish, so this patch should give an indication that something is wrong in those cases as well. If anyone is interested, I've attached the patch. This patch is just an example of how such monitoring can be implemented. This is not production ready code. Performance impacts have not been considered! The patch is for the following class: org.apache.tomcat.util.threads.ThreadPoolExecutor It uses 2 system properties for configuration: org.apache.catalina.threadmonitor.interval (in seconds. Default 120) org.apache.catalina.threadmonitor.threshold (in seconds. Default 600) The monitoring is enabled by default, but can be disabled by setting the "interval" to a negative value or 0. As stated, it only monitors managed threads created by Tomcat's HTTP/AJP thread pools. Monitoring all JVM threads is also an option, but may require more code and might lead to many false positives..? The most important thing is to detect problems with stuck HTTP/AJP threads. TODOs: -Should the thread monitor properties be moved to server.xml instead of being System properties? -Add JMX notification when hung thread detected/hung thread finished I have no problem understanding that such a feature might not be seen as useful for everybody. We will continue to patch our own version of Tomcat anyway :) -- Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
