One of our hosting customers is a medical practice using a commercial EMR
running on tomcat+mysql. It has operated well for over a year, but users have
suddenly begun experiencing slowness for about an hour at the same time every
day. During the slow times, we've done all the usual troubleshooting to catch
the problem in the act. The servers have plenty of power and are not
overworked. There are no slow database queries. Network connectivity is solid.
Tomcat has plenty of memory. The numbers of database connections, threads,
questions, queries, etc., remain steady, without spikes. There is no unusual
disk latency. We have not found any maintenance tasks running during that
timeframe.
The customer has another load-balanced tomcat instance on a different physical
server, and the problem happens on that one, too. The servers were upgraded
with a new kernel and packages on 4/5/24, but the issue did not appear until
5/6/24. The vendor enabled a new feature in the customer's software, and the
problem appeared the next day, but they subsequently disabled the feature, and
(reportedly) the problem did not go away. It is worth mentioning that the
servers are multi-tenanted, with other customers running the same medical
application, but the others do not experience the slowdowns, even though they
are on the same servers.
There are no unusual errors in the tomcat or database server logs, EXCEPT this
one: Java.sql.DriverManager.getConnection
During the periods of slowness, we see lots of those errors along with a large
spike in the number of stuck tomcat threads (from 1 or 2 to as high as 100). It
seems obvious that the threads are stuck because tomcat is waiting on a
connection to the database. However, tcpdump shows that connectivity to the
database is perfect at the network and application layers. There are no
unanswered SYNs, no retransmissions, no half-open connections, no failures to
allocate TCP ports, no conntrack messages, and no other indications of system
resource exhaustion. Every time tomcat requests a connection to the DB, it
completes in less than 1 ms. Ten thousand connection attempts completed
successfully in about 15 seconds, with zero failures.
We are forced to conclude that some database connection requests are being
initiated but are not being sent on the wire. The problem seems to be in the
interaction between tomcat and the database driver, or in the driver itself.
Unfortunately, the application vendor is taking the "it's your infrastructure"
position without providing any evidence or offering suggestions for
configuration changes, other than to deploy more tomcat instances, which is
just shooting in the dark. They don't know why the software is throwing
java.sql.DriverManager.getConnection errors (even though it's their code), and
they've relegated the investigation to us.
Any advice from the community would be greatly appreciated.
RHEL 8.9, kernel 4.18.0-513.18.1.el8_9.x86_64
Apache Tomcat/9.0.80, JVM 1.8.0_372-b07
(The tomcat and JVM versions are the ones recommended by the vendor.)
We're standing by to provide whatever other information the community may need.
Thanks tons!
-Eric
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
