Re: Monitoring Virtual Threads via JMX / MBeans in Tomcat
Dear Tomcat Users, I hope this message finds you well. As per your previous email, we attempted to fetch the virtual thread count from the keepAliveCount attribute in the Catalina.ThreadPool MBean. For context, here is the setup we used: * We created a sample Spring Boot application that continuously creates virtual threads in a loop. * The application was deployed in the TOMCAT_LOCATION/webapps directory and started on localhost. In Apache Tomcat 10.1.36, we added the following configuration to the server.xml file to enable virtual threads: Its observed in the logs that virtual threads were being created as expected. However, we noticed that the keepAliveCount attribute in the Catalina.ThreadPool MBean is showing a value of 0, even though virtual threads are being spawned. It seems that the keepAliveCount attribute does not provide a valid value for counting the virtual threads. We were wondering if this is expected behavior, or if there is a different way to monitor the virtual threads created in Tomcat. We would also like to know if there's a way to differentiate between platform threads and virtual threads using any MBean attribute in the Catalina service or elsewhere in Tomcat's MBean architecture. We would greatly appreciate any guidance or insights you can provide regarding this issue. Best Regards, Rose Mary From: Joash Jose Date: Wednesday, 26 March 2025 at 12:48 PM To: Rose Mary P T Subject: Begin forwarded message: From: Mark Thomas Subject: [EXTERNAL] Re: Monitoring Virtual Threads via JMX / MBeans in Tomcat Date: 6 March 2025 at 2:08:43 PM IST To: Reply-To: "Tomcat Users List" On 06/03/2025 06:29, Joash Jose wrote: Dear Apache Tomcat Support Team, I hope this message finds you well. I am writing to inquire whether Apache Tomcat (tomacat version is 10.1.33 running on Java 21) exposes virtual thread metrics through JMX / MBeans. Specifically: Virtual Thread Visibility: Does Tomcat provide MBeans (e.g., under Catalina:type=Executor) to monitor virtual thread usage, such as active virtual thread counts, creation rate, or parking states? No with a few caveats - see below). If not, are there plans to add such metrics in future releases? No. Or any alternate ways with which we can monitor this from tomcat side.? (connectionCount - keepAliveCount) should be a reasonable estimate of the virtual threads currently being used. If the requests are synchronous (and there is little point using virtual threads with async requests) then the creation rate is derivable from the requestCount in the GlobalRequestProcessor. There is no information on parking states. Configuration Clarification: We have configured Tomcat 10.1.33 with, The element isn't being used. Why configure it? While this works, the existing ThreadPool MBeans show currentThreadsBusy=-1 and maxThreads=200 (incorrect in case of virtual threads). Is this expected behavior? Yes. JVM vs. Tomcat Metrics: We observe that the JVM’s java.lang.management.ThreadMXBean includes virtual threads, but this aggregates data across all applications on the JVM. Does Tomcat offer a way to isolate virtual thread metrics specific to Tomcat? No, because the JVM doesn't provide a mechanism to have multiple pools/groups/anything of virtual threads. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Begin forwarded message: From: Joash Jose Subject: Monitoring Virtual Threads via JMX / MBeans in Tomcat Date: 6 March 2025 at 11:59:29 AM IST To: , Dear Apache Tomcat Support Team, I hope this message finds you well. I am writing to inquire whether Apache Tomcat (tomacat version is 10.1.33 running on Java 21) exposes virtual thread metrics through JMX / MBeans. Specifically: Virtual Thread Visibility: Does Tomcat provide MBeans (e.g., under Catalina:type=Executor) to monitor virtual thread usage, such as active virtual thread counts, creation rate, or parking states? If not, are there plans to add such metrics in future releases? Or any alternate ways with which we can monitor this from tomcat side.? Configuration Clarification: We have configured Tomcat 10.1.33 with, While this works, the existing ThreadPool MBeans show currentThreadsBusy=-1 and maxThreads=200 (incorrect in case of virtual threads). Is this expected behavior? JVM vs. Tomcat Metrics: We observe that the JVM’s java.lang.management.ThreadMXBean includes virtual threads, but this aggregates data across all applications on the JVM. Does Tomcat offer a way to isolate virtual thread metrics specific to Tomcat? Regards, Joash Begin forwarded message: From: Joash Jose Subject: [EXTERNAL] Monitoring Virtual Threads via JMX / MBeans in Tomcat Date: 6 March 2025 at 11:59:40 AM IST To: "d...@tomcat.apache.org" , "users@tomcat.apache.org" Reply-To: "Tomcat Users List" Dear Ap
Re: NIO Thread Madness
William, On 3/25/25 2:51 PM, William Crowell wrote: Mark, I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following: … accessToUnderlyingConnectionAllowed=”true” maxIdle=”100” maxWaitMillis=”1” minIdle=”25” validationQuery=”select 1 from dual” url=”blah blah blah” maxTotal=”true” ^ This is typically a number, not a true/false value. logAbandoned=”true” removeAbandonedOnBorrow=”true” removeAbandonedTimeout=”900” removeAbandonedOnMaintenance=”true” timeBetweenEvictionRunsMillis=”30” When the database pool dries up, your application will basically stop. Setting removeAbandoned as you have is good, but with those long long timeouts, they aren't doing any good. I might set up a connection pool for "short queries" (like logging-in, etc.) and then set up a separate pool for much longer queries. Also, you might want to review your use of connections, etc. to ensure that you are always closing everything in finally blocks. Resource management with JDBC can be tedious and if you don't do it right, you can leak connections from your pool very easily. -chris From: Mark Thomas Date: Tuesday, March 25, 2025 at 1:13 PM To: users@tomcat.apache.org Subject: Re: NIO Thread Madness On 25/03/2025 12:33, William Crowell wrote: Mark, I believe there is a proxy involved here that does TLS decrypt, but I noticed they had the redirectPort on the 8080 connector set to 8443. When you try to hit Tomcat directly over port 8080 using HTTP it is hung. Hmm. Both the Acceptor thread and the Poller thread are running and appear to be in states that would enable new requests to be processed. There isn't logging in those two components for normal operations. I assume because of performance concerns. Do you have any thread dumps from when you have clients attempting new requests that are hanging? I'm wondering if processing is hanging somewhere in Tomcat / the application during request processing. I would probably be thinking about enabling remote debugging and connecting a debugger the next time it goes wrong and tracing the progress of a new request. But I accept that may not be practical for a production system. It seems unlikely that the new requests are hanging before they reach the Tomcat code but that seems unlikely. Other things to try: netstat may shed some light on the current state of Tomcat and database connections. Wireshark or similar on the client side might also provide some clues. I am wondering if Tomcat / Windows is responding at all, dropping the connection, sending a reset, etc. There might be some clues in the exceptions the exceptions that occur just before the problem starts. Can you share any of them? Mark Regards, William Crowell From: Mark Thomas Date: Tuesday, March 25, 2025 at 8:27 AM To: users@tomcat.apache.org Subject: Re: NIO Thread Madness On 25/03/2025 11:24, William Crowell wrote: Chris, Looking at JMX is the next step. I make a request and Tomcat never returns, and I do not get a “connection refused”. It just sits and hangs. Looking that the thread dump you sent me privately now. Which port/protocol are you using to connect to Tomcat? HTTP and 8080? Are you connecting directly to Tomcat or is there a proxy involved at all? Mark Regards, William Crowell From: Christopher Schultz Date: Tuesday, March 25, 2025 at 7:20 AM To: users@tomcat.apache.org Subject: Re: NIO Thread Madness William, On 3/24/25 2:56 PM, William Crowell wrote: I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap. I have an application deployed on this server that is hitting an Oracle database server. I have noticed the server stops accepting requests after about 8-12 hours of uptime. In JProfiler you can tell when this is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked state while querying the database. After this situation clears up, the NIO thread pool grows slightly by about 15-20 threads, and then the application server stops serving requests. I looked at the GC log, and it looks completely healthy, and we are not even close to our max heap. Metaspace size is not configured, but it looks fine from the GC logs. There is no crash file or core dump produced. I do notice some Oracle exceptions in the logs when this happens. We do have about 1000 max connections defined on the Oracle database (which is too many). Are you able to use JMX or similar to see how many used database connections you have? When you say that Tomcat does not accept requests, do you mean that you make a request and it never returns, or do you mean that you get a "connection refused" or something similar? I have my thread pool defined as follows in server.xml: … H
Re: NIO Thread Madness
Chris, That maxTotal was a typo due to trying to copy the config from a screenshot. I agree with your assessment. I think there are 2 situations going on. One is that the code may not be properly closing connections, and the connection pool is not properly configured. Regards, William Crowell From: Christopher Schultz Date: Wednesday, March 26, 2025 at 6:57 PM To: users@tomcat.apache.org Subject: Re: NIO Thread Madness William, On 3/25/25 2:51 PM, William Crowell wrote: > Mark, > > I think we might have found something. I think the DBCP2 connection pool is > returning stale connections from Oracle. There is no firewall between Tomcat > and Oracle, but I looked at the context.xml and found the following: > > … > accessToUnderlyingConnectionAllowed=”true” > maxIdle=”100” > maxWaitMillis=”1” > minIdle=”25” > validationQuery=”select 1 from dual” > url=”blah blah blah” > maxTotal=”true” ^ This is typically a number, not a true/false value. > logAbandoned=”true” > removeAbandonedOnBorrow=”true” > removeAbandonedTimeout=”900” > removeAbandonedOnMaintenance=”true” > timeBetweenEvictionRunsMillis=”30” minEvictableIdleTimeMillis=”90” > The defaults for timeBetweenEvictionRunsMillis is 5 seconds and > minEvictableIdleTimeMillis is 60 seconds, and I also see > removeAbandonedTimeout is set to 15 minutes. Some of the queries to the > database can run over 10 minutes. Sounds like an opportunity to recode this > asynchronously. > > Why this would cause Tomcat to go dark is beyond me. I will work on getting > new thread dumps and stack traces. Most of the long stack traces point to > issues with the database, and they are being sent over as screen shots. I’ll > see what I can do to work around that. When the database pool dries up, your application will basically stop. Setting removeAbandoned as you have is good, but with those long long timeouts, they aren't doing any good. I might set up a connection pool for "short queries" (like logging-in, etc.) and then set up a separate pool for much longer queries. Also, you might want to review your use of connections, etc. to ensure that you are always closing everything in finally blocks. Resource management with JDBC can be tedious and if you don't do it right, you can leak connections from your pool very easily. -chris > From: Mark Thomas > Date: Tuesday, March 25, 2025 at 1:13 PM > To: users@tomcat.apache.org > Subject: Re: NIO Thread Madness > On 25/03/2025 12:33, William Crowell wrote: >> Mark, >> >> I believe there is a proxy involved here that does TLS decrypt, but I >> noticed they had the redirectPort on the 8080 connector set to 8443. When >> you try to hit Tomcat directly over port 8080 using HTTP it is hung. > > Hmm. Both the Acceptor thread and the Poller thread are running and > appear to be in states that would enable new requests to be processed. > > There isn't logging in those two components for normal operations. I > assume because of performance concerns. > > Do you have any thread dumps from when you have clients attempting new > requests that are hanging? I'm wondering if processing is hanging > somewhere in Tomcat / the application during request processing. > > I would probably be thinking about enabling remote debugging and > connecting a debugger the next time it goes wrong and tracing the > progress of a new request. But I accept that may not be practical for a > production system. > > It seems unlikely that the new requests are hanging before they reach > the Tomcat code but that seems unlikely. > > Other things to try: > > netstat may shed some light on the current state of Tomcat and database > connections. > > Wireshark or similar on the client side might also provide some clues. > > I am wondering if Tomcat / Windows is responding at all, dropping the > connection, sending a reset, etc. > > There might be some clues in the exceptions the exceptions that occur > just before the problem starts. Can you share any of them? > > Mark > >> >> Regards, >> >> William Crowell >> >> From: Mark Thomas >> Date: Tuesday, March 25, 2025 at 8:27 AM >> To: users@tomcat.apache.org >> Subject: Re: NIO Thread Madness >> On 25/03/2025 11:24, William Crowell wrote: >>> Chris, >>> >>> Looking at JMX is the next step. I make a request and Tomcat never >>> returns, and I do not get a “connection refused”. It just sits and hangs. >> >> Looking that the thread dump you sent me privately now. >> >> Which port/protocol are you using to connect to Tomcat? HTTP and 8080? >> >> Are you connecting directly to Tomcat or is there a proxy involved at all? >> >> Mark >> >>> >>> Regards, >>> >>> William Crowell >>> >>> From: Christopher Schultz >>> Date: Tuesday, March 25, 2025 at 7:20 AM >>> To: users@tomcat.apache.org >>> Subject: Re: NIO Thread Madness >>> William, >>> >>> On 3/24/25 2:56 PM, William Crowell wrote: I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle JDK 1.8.0_371-b11
Re: Monitoring Virtual Threads via JMX / MBeans in Tomcat
Dear Tomcat Users, I hope this message finds you well. As per your previous email, we attempted to fetch the virtual thread count from the keepAliveCount attribute in the Catalina.ThreadPool MBean. For context, here is the setup we used: * We created a sample Spring Boot application that continuously creates virtual threads in a loop. * The application was deployed in the TOMCAT_LOCATION/webapps directory and started on localhost. In Apache Tomcat 10.1.36, we added the following configuration to the server.xml file to enable virtual threads: Its observed in the logs that virtual threads were being created as expected. However, we noticed that the keepAliveCount attribute in the Catalina.ThreadPool MBean is showing a value of 0, even though virtual threads are being spawned. It seems that the keepAliveCount attribute does not provide a valid value for counting the virtual threads. We were wondering if this is expected behavior, or if there is a different way to monitor the virtual threads created in Tomcat. We would also like to know if there's a way to differentiate between platform threads and virtual threads using any MBean attribute in the Catalina service or elsewhere in Tomcat's MBean architecture. We would greatly appreciate any guidance or insights you can provide regarding this issue. Best Regards, Rose Mary From: Joash Jose Date: Wednesday, 26 March 2025 at 12:48 PM To: Rose Mary P T Subject: Begin forwarded message: From: Mark Thomas Subject: [EXTERNAL] Re: Monitoring Virtual Threads via JMX / MBeans in Tomcat Date: 6 March 2025 at 2:08:43 PM IST To: Reply-To: "Tomcat Users List" On 06/03/2025 06:29, Joash Jose wrote: Dear Apache Tomcat Support Team, I hope this message finds you well. I am writing to inquire whether Apache Tomcat (tomacat version is 10.1.33 running on Java 21) exposes virtual thread metrics through JMX / MBeans. Specifically: Virtual Thread Visibility: Does Tomcat provide MBeans (e.g., under Catalina:type=Executor) to monitor virtual thread usage, such as active virtual thread counts, creation rate, or parking states? No with a few caveats - see below). If not, are there plans to add such metrics in future releases? No. Or any alternate ways with which we can monitor this from tomcat side.? (connectionCount - keepAliveCount) should be a reasonable estimate of the virtual threads currently being used. If the requests are synchronous (and there is little point using virtual threads with async requests) then the creation rate is derivable from the requestCount in the GlobalRequestProcessor. There is no information on parking states. Configuration Clarification: We have configured Tomcat 10.1.33 with, The element isn't being used. Why configure it? While this works, the existing ThreadPool MBeans show currentThreadsBusy=-1 and maxThreads=200 (incorrect in case of virtual threads). Is this expected behavior? Yes. JVM vs. Tomcat Metrics: We observe that the JVM’s java.lang.management.ThreadMXBean includes virtual threads, but this aggregates data across all applications on the JVM. Does Tomcat offer a way to isolate virtual thread metrics specific to Tomcat? No, because the JVM doesn't provide a mechanism to have multiple pools/groups/anything of virtual threads. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Begin forwarded message: From: Joash Jose Subject: Monitoring Virtual Threads via JMX / MBeans in Tomcat Date: 6 March 2025 at 11:59:29 AM IST To: , Dear Apache Tomcat Support Team, I hope this message finds you well. I am writing to inquire whether Apache Tomcat (tomacat version is 10.1.33 running on Java 21) exposes virtual thread metrics through JMX / MBeans. Specifically: Virtual Thread Visibility: Does Tomcat provide MBeans (e.g., under Catalina:type=Executor) to monitor virtual thread usage, such as active virtual thread counts, creation rate, or parking states? If not, are there plans to add such metrics in future releases? Or any alternate ways with which we can monitor this from tomcat side.? Configuration Clarification: We have configured Tomcat 10.1.33 with, While this works, the existing ThreadPool MBeans show currentThreadsBusy=-1 and maxThreads=200 (incorrect in case of virtual threads). Is this expected behavior? JVM vs. Tomcat Metrics: We observe that the JVM’s java.lang.management.ThreadMXBean includes virtual threads, but this aggregates data across all applications on the JVM. Does Tomcat offer a way to isolate virtual thread metrics specific to Tomcat? Regards, Joash Begin forwarded message: From: Joash Jose Subject: [EXTERNAL] Monitoring Virtual Threads via JMX / MBeans in Tomcat Date: 6 March 2025 at 11:59:40 AM IST To: "d...@tomcat.apache.org" , "users@tomcat.apache.org" Reply-To: "Tomcat Users List" Dear Ap
