https://bz.apache.org/bugzilla/show_bug.cgi?id=59737
Bug ID: 59737
Summary: Tomcat endpoints are becoming extremely unresponsive
Product: Tomcat 8
Version: 8.0.35
Hardware: PC
OS: Windows NT
Status: NEW
Severity: blocker
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: rcraigly...@gmail.com
We run embedded tomcat on version 8, and for some reason are consistently
seeing extreme slowness across all Tomcat endpoints at very consistent
intervals of three hours. Once a site gets into the slow state, it is never
able to recover, and stays unresponsive (requests take tens of minutes to
hours) until the service is manually restarted.
There are no resource issues that I've been able to detect (heap seems fine, no
apparent memory leaks, cpu is fine, network/db connections aren't exhausted,
etc). Tomcat does seem to receive the requests, but for some reason does not
seem to be processing them.
There is an exception that occurs right around the time the service goes into a
bad state, which is the reason I believe this to be a Tomcat issue. The stack
trace is as follows:
Exception in thread "mc-26" java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown
Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Having done some additional testing, I can elaborate on the state of things
when this exception occurs:
* The ReentrantLock that is throwing the exception is meant to throw the
IllegalMonitorStateException when a thread other than the thread that created
the lock tries to release it. Interestingly, when tryRelease() is called, the
owning thread is actually null, which means that the lock isn't currently taken
or owned by anyone.
* The lock's state is 0, which is consistent with the lock's owning thread
being null.
* The tryRelease funtion takes an int argument "releases", which is 1 (you'd
find that in the stacktrace anyway, since it's passed in as a constant further
up, but mentioning it might save some time).
I realize that the LinkedBlockingQueue and ReentrantLock are java core
concurrency libraries, but it seems like Tomcat is getting into a bad state
once the error occurs, and is unable to exit the bad state. The executing
thread that gets this exception dies, and almost all other threads end up
staying almost all the time in an unsafe parked state. The
IllegalMonitorStateException also generally occurs in several threads after
it's shown up for the first time.
I'm not sure how to describe how to reproduce this issue, other than saying
that everyone at my company with our service installed experiences it very
reliably every three hours. We've been thus far unable to determine what causes
it, however. My personal theory is that somehow the TaskQueue is getting into a
state where it can only rarely give tasks to the executor threads, but I don't
know what causes things to enter that state. From what we can tell, it does
seem to be related to our Jersey endpoints.
If you need any additional information, just let me know and I'll be happy to
provide anything that might be useful. I've been getting most of my information
from thread/heap dumps as well as modifying local versions of Tomcat to provide
additional logging.
Most of our services are running Tomcat 8.0.32, but seem to still exhibit the
problem on versions at least as early as 8.0.15 and as late as 8.0.34.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
