from:"John Williams"

Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-28 Thread John Williams


Hi Everyone,

I am running Apache Tomcat 10.1.30 for a web application and notice ~950 
timed_waiting threads.

The stack trace for these threads is below:

java.base@17.0.12/jdk.internal.misc.Unsafe.park(Native
 Method);
java.base@17.0.12/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252);
java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674);
java.base@17.0.12/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1113);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1175);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659);
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63);
java.base@17.0.12/java.lang.Thread.run(Thread.java:840);

I am using an executor with maxThreads set to 1500 and minSpareThreads set to 
64.

I have not seen this issue with Tomcat 10.0.

What could be the issue and how to resolve it? Any help is much appreciated.

Regards,

John

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Chris,

Sorry - didn't realize that images would be stripped off. 

The number of TIMED_WAITING threads starts at 150 or so and every hour 
increases by 10.

Below is an example of where the threads are stuck (by looking at the JVM 
stack):

https-jsse-nio2-443-exec-90
TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@48986fd7
jdk.internal.misc.Unsafe.park(Native Method);
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252);   
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674);
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99); 
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33); 
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1113);
 
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1175);
 
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659);
 
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63);
 
java.base@17.0.12/java.lang.Thread.run(Thread.java:840);

The connector configuration is:





Regards,


-Original Message-
From: Christopher Schultz  
Sent: Monday, September 30, 2024 10:38 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


John,

On 9/30/24 10:30, John Williams wrote:
> The issue is that the number of threads in the TIMED_WAIT state keeps 
> increasing over time. Starts at 150 and keeps growing by 10 every hour.
> Once it reaches close to the maxThreads setting we see connection drops.
>
> This application was working fine with Tomcat 10.0 and JDK 12. Unless 
> the application was seeing load, threads would not be created in 
> Tomcat's JVM.
>
> You can also see the thread stacks in this image.

This list strips out image. Can you post text?

Please post your  configuration taking care to remove any secrets.

Thanks,
-chris

> -Original Message-
> From: Christopher Schultz 
> Sent: Monday, September 30, 2024 10:20 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
>
> [You don't often get email from ch...@christopherschultz.net 
> <mailto:ch...@christopherschultz.net>. Learn why this is important at 
> https://aka.ms/LearnAboutSenderIdentification 
> <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka
> .ms%2F&data=05%7C02%7Csrinivas%40eginnovations.com%7C5c93ec3de0884e786
> 7d708dce15d7e73%7C62c210cb17214d259b6b0c95ceb472ce%7C0%7C0%7C638633038
> 912339442%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6q8HQHLjceSHYLqEnLN9
> ue%2Fpa3zz5cougsajpZIsjDw%3D&reserved=0
> LearnAboutSenderIdentification> ]
>
> CAUTION: This email originated from outside of the organization. Do 
> not click links or open attachments unless you recognize the sender 
> and know the content is safe.
>
> John,
>
> On 9/28/24 05:38, John Williams wrote:
>
>  >
>
>  > Hi Everyone,
>
>  >
>
>  > I am running Apache Tomcat 10.1.30 for a web application and notice
> ~950 timed_waiting threads.
>
>  >
>
>  > The stack trace for these threads is below:
>
>  >
>
>  > java.base@17.0.12/
> jdk.internal.misc.Unsafe.park(Native<mailto:java.bas
> <mailto:java.base@17.0.12/
> jdk.internal.misc.Unsafe.park(Native%3cmailto:java.bas>
>
>  > e@17.0.12/jdk.internal.misc.Unsafe.park(Native <mailto:e@17.0.12/ 
> jdk.internal.misc.Unsafe.park(Native>> Method);
>
>  > java.base@17.0.12/
> java.util.concurrent.locks.LockSupport.parkNanos(Loc
> <mailto:java.base@17.0.12/
> java.util.concurrent.locks.LockSupport.parkNanos(Loc>
>
>  > 
> kSupport.java:252)<mailto:java.base@17.0.12/java.util.concurrent.locks
>
>  > .LockSupport.parkNanos(LockSupport.java:252)>;
>
>  > java.base@17.0.12/
> java.util.concurrent.locks.AbstractQueuedSynchronize
> <mailto:java.base@17.0.12/
> java.util.concurrent.locks.AbstractQueuedSynchronize>
>
>  > 
> r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)
>  > 
> lto:java.base@17.0.12/java.util.concurrent.locks.Ab

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Hi Chris,

I had an executor defined before and it had the exact same behavior/problem. 
Moved to the below model for the connector only after that. 

Regards,

John

-Original Message-
From: Christopher Schultz  
Sent: Monday, September 30, 2024 1:11 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


John,

On 9/30/24 10:53, John Williams wrote:
> Sorry - didn't realize that images would be stripped off.
>
> The number of TIMED_WAITING threads starts at 150 or so and every hour 
> increases by 10.
>
> Below is an example of where the threads are stuck (by looking at the JVM 
> stack):
>
> https-jsse-nio2-443-exec-90
> TIMED_WAITING on 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@
> 48986fd7 jdk.internal.misc.Unsafe.park(Native Method); 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
> ; 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
> awaitNanos(AbstractQueuedSynchronizer.java:1674);
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java
> :460); 
> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP
> oolExecutor.java:1113); 
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa
> dPoolExecutor.java:1175); 
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre
> adPoolExecutor.java:659); 
> app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
> skThread.java:63); 
> java.base@17.0.12/java.lang.Thread.run(Thread.java:840);
>
> The connector configuration is:
>
> port="443" maxThreads="1536" minSpareThreads="64"
>   
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation" 
> scheme="https" secure="true" SSLEnabled="true"
>   enableLookups="false" acceptCount="12000" 
> connectionTimeout="6" socket.rxBufSize="131072" socket.txBufSize="131072" 
> maxHttpHeaderSize="131072" maxPostSize="-1"  maxKeepAliveRequests="1"  
> maxConnections="1536"  URIEncoding="UTF-8" useSendfile="false" 
> tcpNoDelay="true" compression="force" 
> compressibleMimeType="font/woff,font/woff2,text/html,text/xml,text/plain,application/x-java-applet,application/octet-stream,application/xml,text/javascript,text/css,image/png,image/bmp,image/jpeg,image/gif,application/pdf,application/x-javascript,application/javascript,application/json,application/x-shockwave-flash,application/xhtml+xml,application/xml+xhtml"
>  noCompressionUserAgents="gozilla, traviata">
>ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
>  certificateVerification="none" protocols="TLSv1.2" 
> sslProtocol="TLS"> certificateKeyAlias="..." certificateKeystorePassword="..." />
>   

You are not explicitly using an . When you configure using 
 only, Tomcat will never retire the threads from the thread-pool. 
This is mostly historic and while it could be improved, it is better to switch 
to a mode modern configuration.

If you use  and:



Then you should see the number of threads fluxuate based upon the request 
volume.

I'm not sure why you were seeing different behavior with Tomcat 10.0.
IIRC this behavior has been consistent as long as  and  
have been separate configuration elements.

-chris

> -Original Message-
> From: Christopher Schultz 
> Sent: Monday, September 30, 2024 10:38 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
>
> [You don't often get email from ch...@christopherschultz.net. Learn 
> why this is important at https://aka.ms/LearnAboutSenderIdentification 
> ]
>
> CAUTION: This email originated from outside of the organization. Do

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Hi Chuck,

No - I cannot see any other thread stuck on DB/external resources. The 
application functions as expected, just that I see these threads increasing 
over time. 
The problem was 1st noticed in Tomcat 10.1.26/OpenJRE 17.0.5 and I've tried 
moving to Tomcat 10.1.30/OpenJRE 17.0.12 with no change.

Regards,

-Original Message-
From: Chuck Caldarale  
Sent: Monday, September 30, 2024 1:26 PM
To: Tomcat Users List 
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from n82...@gmail.com. Learn why this is important 
at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

> On Sep 30, 2024, at 12:15, John Williams 
>  wrote:
>
> I had an executor defined before and it had the exact same behavior/problem. 
> Moved to the below model for the connector only after that.

The OP didn't mention the real problem in his original message; it only showed 
up as almost an afterthought in the second posting with the unusable 
attachments:

"Once it reaches close to the maxThreads setting we see connection drops."

I wonder if there's something else blocking the application, such as stuck DB 
operations or other delayed access to an external resource.

  - Chuck

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Chris,



Thanks for your response.



The issue is that the number of threads in the TIMED_WAIT state keeps 
increasing over time. Starts at 150 and keeps growing by 10 every hour. Once it 
reaches close to the maxThreads setting we see connection drops.

This application was working fine with Tomcat 10.0 and JDK 12. Unless the 
application was seeing load, threads would not be created in Tomcat's JVM.



[cid:image003.jpg@01DB1323.CCF9FD00]





You can also see the thread stacks in this image.

[cid:image004.png@01DB1323.CCF9FD00]



Thanks,



-Original Message-
From: Christopher Schultz 
Sent: Monday, September 30, 2024 10:20 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads



[You don't often get email from 
ch...@christopherschultz.net<mailto:ch...@christopherschultz.net>. Learn why 
this is important at https://aka.ms/LearnAboutSenderIdentification ]



CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.





John,



On 9/28/24 05:38, John Williams wrote:

>

> Hi Everyone,

>

> I am running Apache Tomcat 10.1.30 for a web application and notice ~950 
> timed_waiting threads.

>

> The stack trace for these threads is below:

>

> java.base@17.0.12/jdk.internal.misc.Unsafe.park(Native<mailto:java.bas<mailto:java.base@17.0.12/jdk.internal.misc.Unsafe.park(Native%3cmailto:java.bas>

> e@17.0.12/jdk.internal.misc.Unsafe.park(Native<mailto:e@17.0.12/jdk.internal.misc.Unsafe.park(Native>>
>  Method);

> java.base@17.0.12/java.util.concurrent.locks.LockSupport.parkNanos(Loc<mailto:java.base@17.0.12/java.util.concurrent.locks.LockSupport.parkNanos(Loc>

> kSupport.java:252)<mailto:java.base@17.0.12/java.util.concurrent.locks

> .LockSupport.parkNanos(LockSupport.java:252)>;

> java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchronize<mailto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchronize>

> r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674) lto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchro

> nizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)

> >;

> java.base@17.0.12/java.util.concurrent.LinkedBlockingQueue.poll(Linked<mailto:java.base@17.0.12/java.util.concurrent.LinkedBlockingQueue.poll(Linked>

> BlockingQueue.java:460)<mailto:java.base@17.0.12/java.util.concurrent.

> LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460)>;

> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);

> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);

> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP

> oolExecutor.java:1113);

> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa

> dPoolExecutor.java:1175);

> app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre

> adPoolExecutor.java:659);

> app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta

> skThread.java:63);

> java.base@17.0.12/java.lang.Thread.run(Thread.java:840)<mailto:java.ba<mailto:java.base@17.0.12/java.lang.Thread.run(Thread.java:840)%3cmailto:java.ba>

> se@17.0.12/java.lang.Thread.run(Thread.java:840)<mailto:se@17.0.12/java.lang.Thread.run(Thread.java:840)>>;

>

> I am using an executor with maxThreads set to 1500 and minSpareThreads set to 
> 64.

>

> I have not seen this issue with Tomcat 10.0.

>

> What could be the issue and how to resolve it? Any help is much appreciated.



This looks like you have a large number of threads waiting for work.

That seems normal to me if you have a max thread pool of 1500 threads.



What's the problem?



-chris





-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org>

For additional commands, e-mail: 
users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Chris,

maxThreads has been set after reviewing the workload. We used a multiple of 64.

The connections are from bots and are unlikely to be pipelined. Connections can 
be bursty - hence, a bigger acceptCount.

Will revert to 10.0 and see. 

Regards,

John

-Original Message-
From: Christopher Schultz  
Sent: Monday, September 30, 2024 5:39 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

John,

On 9/30/24 13:34, John Williams wrote:
> No - I cannot see any other thread stuck on DB/external resources. The 
> application functions as expected, just that I see these threads increasing 
> over time.
> The problem was 1st noticed in Tomcat 10.1.26/OpenJRE 17.0.5 and I've tried 
> moving to Tomcat 10.1.30/OpenJRE 17.0.12 with no change.

If you go back to 10.0 does it still behave as expected?

You have a bit of a weird setup IMHO.

>  maxThreads="1536"

This is a very strange number. Why not 1537?

 >   minSpareThreads="64"

This seems reasonable, but also kind of low given:

>   acceptCount="12000"

So you are willing to accept HUGE numbers of connections and then let them sit 
in the TCP/IP stack's accept queue waiting for a processing thread. Why?

>   connectionTimeout="6"

Ouch. So you will accept huge numbers of connections and then allow them to sit 
there not making any request? This seems like a recipe for disaster.

>   socket.rxBufSize="131072"
 >   socket.txBufSize="131072"

Tweaking the socket buffer sizes is either an indication of an insanely 
well-tuned performance setup or that you don't know what you are doing.
Given the other settings, I'm inclined toward the latter.

 >   maxKeepAliveRequests="1"

Disabling pipelining? Definitely not expecting a high performance service, or 
you are very sure that no client will ever make more than one request.

 >  maxConnections="1536"

You don't want this, especially for NIO. There is no reason to reduce this from 
the default, although your other settings make the non-blocking nature of the 
NIO connector moot (e.g. pipelining is disabled).

Why do you have this collection of odd settings?

-chris

> -Original Message-
> From: Chuck Caldarale 
> Sent: Monday, September 30, 2024 1:26 PM
> To: Tomcat Users List 
> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
>
> [You don't often get email from n82...@gmail.com. Learn why this is 
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.
>
>
>> On Sep 30, 2024, at 12:15, John Williams 
>>  wrote:
>>
>> I had an executor defined before and it had the exact same behavior/problem. 
>> Moved to the below model for the connector only after that.
>
>
> The OP didn't mention the real problem in his original message; it only 
> showed up as almost an afterthought in the second posting with the unusable 
> attachments:
>
> "Once it reaches close to the maxThreads setting we see connection drops."
>
> I wonder if there's something else blocking the application, such as stuck DB 
> operations or other delayed access to an external resource.
>
>- Chuck
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

6 matches

Site Navigation

Mail list logo

Footer information