date:20240930

Re: Setting Transfer-Encoding: chunked

2024-09-30 Thread Mark Thomas


On 30/09/2024 07:37, Lazar Kirchev wrote:

Hello,

Tomcat automatically adds header Transfer-Encoding: chunked if on http 1.1,
the response code supports body and there is no Connection: Close header
(Tomcat 9's code -
https://github.com/apache/tomcat/blob/372f3cefe6225b58fcdae7c344d81396b8e08570/java/org/apache/coyote/http11/Http11Processor.java#L935
).
However, if the application has already set this header, then the header
gets duplicated. There is no check if the header is already present. Is
this intended behavior?


Applications should not, under any circumstances, be setting the HTTP 
header "Transfer-Encoding: chunked".


Applications do not have sufficient control over the bytes on the wire 
to manually control chunked encoding.


Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: net::ERR_HTTP2_PROTOCOL_ERROR with 10.1.30

2024-09-30 Thread Mark Thomas


On 30/09/2024 07:38, Ahmed Ashour wrote:

Hi all,
Even though the regression should have been fixed in 10.1.30, our team still 
sees it around once weekly. Twice so far.
With 10.1.29 it was very frequent, that the server can't be used, but with 
10.1.30 it is much less, but sadly it seems on rare occasions to occur.


How rare? Once in how many requests? Can you trigger this via automated 
testing e.g. with wrk?



I understand the difficulty of reproducing it, but a hint about using Chrome 
could be beneficial, as it seems it doesn't happen with Firefox for example.
Remarking the UpgradeProtocol configuration makes the team uses the application 
server again.

Please let me know if anything to be done to help in this regard.


Does setting discardRequestsAndResponses="true" help at all?

Mark




Thanks a lot,Ahmed Ashour



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Chris,

Sorry - didn't realize that images would be stripped off. 

The number of TIMED_WAITING threads starts at 150 or so and every hour 
increases by 10.

Below is an example of where the threads are stuck (by looking at the JVM 
stack):

https-jsse-nio2-443-exec-90
TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@48986fd7
jdk.internal.misc.Unsafe.park(Native Method);
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252);   
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674);
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99); 
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33); 
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1113);
 
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1175);
 
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659);
 
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63);
 
java.base@17.0.12/java.lang.Thread.run(Thread.java:840);

The connector configuration is:





Regards,


-Original Message-
From: Christopher Schultz  
Sent: Monday, September 30, 2024 10:38 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


John,

On 9/30/24 10:30, John Williams wrote:
> The issue is that the number of threads in the TIMED_WAIT state keeps 
> increasing over time. Starts at 150 and keeps growing by 10 every hour.
> Once it reaches close to the maxThreads setting we see connection drops.
>
> This application was working fine with Tomcat 10.0 and JDK 12. Unless 
> the application was seeing load, threads would not be created in 
> Tomcat's JVM.
>
> You can also see the thread stacks in this image.

This list strips out image. Can you post text?

Please post your  configuration taking care to remove any secrets.

Thanks,
-chris

> -Original Message-
> From: Christopher Schultz 
> Sent: Monday, September 30, 2024 10:20 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
>
> [You don't often get email from ch...@christopherschultz.net 
> . Learn why this is important at 
> https://aka.ms/LearnAboutSenderIdentification 
>  .ms%2F&data=05%7C02%7Csrinivas%40eginnovations.com%7C5c93ec3de0884e786
> 7d708dce15d7e73%7C62c210cb17214d259b6b0c95ceb472ce%7C0%7C0%7C638633038
> 912339442%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6q8HQHLjceSHYLqEnLN9
> ue%2Fpa3zz5cougsajpZIsjDw%3D&reserved=0
> LearnAboutSenderIdentification> ]
>
> CAUTION: This email originated from outside of the organization. Do 
> not click links or open attachments unless you recognize the sender 
> and know the content is safe.
>
> John,
>
> On 9/28/24 05:38, John Williams wrote:
>
>  >
>
>  > Hi Everyone,
>
>  >
>
>  > I am running Apache Tomcat 10.1.30 for a web application and notice
> ~950 timed_waiting threads.
>
>  >
>
>  > The stack trace for these threads is below:
>
>  >
>
>  > java.base@17.0.12/
> jdk.internal.misc.Unsafe.park(Native  jdk.internal.misc.Unsafe.park(Native%3cmailto:java.bas>
>
>  > e@17.0.12/jdk.internal.misc.Unsafe.park(Native  jdk.internal.misc.Unsafe.park(Native>> Method);
>
>  > java.base@17.0.12/
> java.util.concurrent.locks.LockSupport.parkNanos(Loc
>  java.util.concurrent.locks.LockSupport.parkNanos(Loc>
>
>  > 
> kSupport.java:252)
>  > .LockSupport.parkNanos(LockSupport.java:252)>;
>
>  > java.base@17.0.12/
> java.util.concurrent.locks.AbstractQueuedSynchronize
>  java.util.concurrent.locks.AbstractQueuedSynchronize>
>
>  > 
> r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)
>  > 
> lto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchro
>
>  > 
> nizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)
>
>  > >;
>
>  > java.base@17.0.12/
> java.util.concurrent.LinkedBlockingQueue.poll(Linked
>  java.util.concurrent.LinkedBlockingQueue.poll(Linked>
>
>  > BlockingQueue.java:460)
>  > LinkedBlockingQueue.pol

Re: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread Christopher Schultz

John,

On 9/30/24 10:53, John Williams wrote:

Sorry - didn't realize that images would be stripped off.

The number of TIMED_WAITING threads starts at 150 or so and every hour 
increases by 10.

Below is an example of where the threads are stuck (by looking at the JVM 
stack):

https-jsse-nio2-443-exec-90
TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@48986fd7
jdk.internal.misc.Unsafe.park(Native Method);
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252);
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674);
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1113);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1175);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659);
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63);
java.base@17.0.12/java.lang.Thread.run(Thread.java:840);

The connector configuration is:

You are not explicitly using an . When you configure using 
 only, Tomcat will never retire the threads from the 
thread-pool. This is mostly historic and while it could be improved, it 
is better to switch to a mode modern configuration.

If you use  and:

Then you should see the number of threads fluxuate based upon the 
request volume.

I'm not sure why you were seeing different behavior with Tomcat 10.0. 
IIRC this behavior has been consistent as long as  and 
 have been separate configuration elements.

-chris

-Original Message-
From: Christopher Schultz 
Sent: Monday, September 30, 2024 10:38 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

John,

On 9/30/24 10:30, John Williams wrote:

The issue is that the number of threads in the TIMED_WAIT state keeps
increasing over time. Starts at 150 and keeps growing by 10 every hour.
Once it reaches close to the maxThreads setting we see connection drops.

This application was working fine with Tomcat 10.0 and JDK 12. Unless
the application was seeing load, threads would not be created in
Tomcat's JVM.

You can also see the thread stacks in this image.

This list strips out image. Can you post text?

Please post your  configuration taking care to remove any secrets.

Thanks,
-chris

-Original Message-
From: Christopher Schultz 
Sent: Monday, September 30, 2024 10:20 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net
. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification
 ]

CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe.

John,

On 9/28/24 05:38, John Williams wrote:

  >

  > Hi Everyone,

  >

  > I am running Apache Tomcat 10.1.30 for a web application and notice
~950 timed_waiting threads.

  >

  > The stack trace for these threads is below:

  >

  > java.base@17.0.12/
jdk.internal.misc.Unsafe.park(Native

  > e@17.0.12/jdk.internal.misc.Unsafe.park(Native > Method);

  > java.base@17.0.12/
java.util.concurrent.locks.LockSupport.parkNanos(Loc

  >
kSupport.java:252) .LockSupport.parkNanos(LockSupport.java:252)>;

  > java.base@17.0.12/
java.util.concurrent.locks.AbstractQueuedSynchronize

  >
r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)
lto:java.base@17.0.12/j

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Hi Chris,

I had an executor defined before and it had the exact same behavior/problem. 
Moved to the below model for the connector only after that. 

Regards,

John

-Original Message-
From: Christopher Schultz  
Sent: Monday, September 30, 2024 1:11 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.


John,

On 9/30/24 10:53, John Williams wrote:
> Sorry - didn't realize that images would be stripped off.
>
> The number of TIMED_WAITING threads starts at 150 or so and every hour 
> increases by 10.
>
> Below is an example of where the threads are stuck (by looking at the JVM 
> stack):
>
> https-jsse-nio2-443-exec-90
> TIMED_WAITING on 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@
> 48986fd7 jdk.internal.misc.Unsafe.park(Native Method); 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
> ; 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
> awaitNanos(AbstractQueuedSynchronizer.java:1674);
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java
> :460); 
> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP
> oolExecutor.java:1113); 
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa
> dPoolExecutor.java:1175); 
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre
> adPoolExecutor.java:659); 
> app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
> skThread.java:63); 
> java.base@17.0.12/java.lang.Thread.run(Thread.java:840);
>
> The connector configuration is:
>
> port="443" maxThreads="1536" minSpareThreads="64"
>   
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation" 
> scheme="https" secure="true" SSLEnabled="true"
>   enableLookups="false" acceptCount="12000" 
> connectionTimeout="6" socket.rxBufSize="131072" socket.txBufSize="131072" 
> maxHttpHeaderSize="131072" maxPostSize="-1"  maxKeepAliveRequests="1"  
> maxConnections="1536"  URIEncoding="UTF-8" useSendfile="false" 
> tcpNoDelay="true" compression="force" 
> compressibleMimeType="font/woff,font/woff2,text/html,text/xml,text/plain,application/x-java-applet,application/octet-stream,application/xml,text/javascript,text/css,image/png,image/bmp,image/jpeg,image/gif,application/pdf,application/x-javascript,application/javascript,application/json,application/x-shockwave-flash,application/xhtml+xml,application/xml+xhtml"
>  noCompressionUserAgents="gozilla, traviata">
>ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
>  certificateVerification="none" protocols="TLSv1.2" 
> sslProtocol="TLS"> certificateKeyAlias="..." certificateKeystorePassword="..." />
>   

You are not explicitly using an . When you configure using 
 only, Tomcat will never retire the threads from the thread-pool. 
This is mostly historic and while it could be improved, it is better to switch 
to a mode modern configuration.

If you use  and:



Then you should see the number of threads fluxuate based upon the request 
volume.

I'm not sure why you were seeing different behavior with Tomcat 10.0.
IIRC this behavior has been consistent as long as  and  
have been separate configuration elements.

-chris

> -Original Message-
> From: Christopher Schultz 
> Sent: Monday, September 30, 2024 10:38 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
>
> [You don't often get email from ch...@christopherschultz.net. Learn 
> why this is important at https://aka.ms/LearnAboutSenderIdentification 
> ]
>
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.
>
>
> John,
>
> On 9/30/24 10:30, John Williams wrote:
>> The issue is that the number of threads in the TIMED_WAIT state keeps 
>> increasing over time. Starts at 150 and keeps growing by 10 every hour.
>> Once it reaches close to the maxThreads setting we see connection drops.
>>
>> This application was working fine with Tomcat 10.0 and JDK 12. Unless 
>> the application was s

Re: [OT] accessing manager app

2024-09-30 Thread Christopher Schultz


Michael,

On 9/30/24 11:41, Michael Osipov wrote:

Chris,

On 2024/09/30 14:33:53 Christopher Schultz wrote:

Michael,

On 9/28/24 13:34, Michael Osipov wrote:

On 2024/09/27 15:14:15 Christopher Schultz wrote:

Sebastian,

On 9/27/24 11:04, Sebastian Trost wrote:

Francesco,

On 26.09.2024 16:12, Francesco Viscomi wrote:

Hi all,
I'm not able to understand why I cannot access to
    http://localhost:8080/manager/html

I've configured the user in tomcat.users.xml:




I'm using tomcat 9; and jdk17;

I've also noted that in my personal pc when try to access manager/html a
pop up ask me to login (in my personal pc it works right)

While when I try to use it in the company pc it gives me 401
unauthorized;
I do not know what I have to modify on chrome to get access in manager
app,
I also use in the company pc Zscaler, but I do not know what I have to
change in it (eventually) in order to access the manager app.

Your corporate browser probably has basic authentication disabled. Check
this site: https://jigsaw.w3.org/HTTP/Basic
If there is no basic authentication popup where you can enter username/
password then this is probably the case.

See: https://answers.microsoft.com/en-us/microsoftedge/forum/all/latest-
version-of-edge-no-longer-shows-basic/3601252b-e56b-46c0-a088-0f6084eabe47


I've really had it with Microsoft deciding that HTTP Basic
authentication is just not okay. They seem to have forgotten that TLS
makes it secure.


The reasoning is never to share a long term secret: your password.


HTTP Digest also requires pre-shared passwords.


There is a subtile difference: the password is never transferred over the wire 
and does not appear on the target server.


While that may be true, it is irrelevant. The 
MD5(username:reaml:password) must be known to the server. That is as 
good as the password in terms of security.


The realm name can never be changed without changing all passwords. The 
algorithm used can never be changed without changing all passwords.


The overwhelming majority of web-based applications use pre-shared 
passwords with FORM-based authentication over TLS. There is zero 
reduction in security when compared to HTTP Digest. In both cases, 
hashes can be stored on the server-side which is of course a best-practice.



HTTP Digest is a nightmare, but they are forcing users onto it.


The key is to use SPNEGO in enterprise environments.


What about non-enterprise environments?


IMHO, this is irrelevant for Microsoft. In enterprise you do have at least 
SPNEGO or even PKI. For non-enterprise I see only Basic as a viable option.


Except that Microsoft is killing it.

At $work, we use WebDAV over TLS with an OpenLDAP back-end for 
authentication. It works great for all employees except those who use 
Windows. It seems like every installation of Windows needs a different 
hack to get HTTP Basic working when connecting to WebDAV.


There is no "Enterprise" here. There are no Domain Controllers here. 
There is no expensive third-party authentication-as-a-service company 
here. There is only a standards-compliant service which is not reliably 
accessible to users on Microsoft Windows.


And it annoys the hell out of me.

-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread Chuck Caldarale


> On Sep 30, 2024, at 12:15, John Williams 
>  wrote:
> 
> I had an executor defined before and it had the exact same behavior/problem. 
> Moved to the below model for the connector only after that. 


The OP didn’t mention the real problem in his original message; it only showed 
up as almost an afterthought in the second posting with the unusable 
attachments:

"Once it reaches close to the maxThreads setting we see connection drops."

I wonder if there’s something else blocking the application, such as stuck DB 
operations or other delayed access to an external resource.

  - Chuck

Re: accessing manager app

2024-09-30 Thread Michael Osipov

Chris,

On 2024/09/30 14:33:53 Christopher Schultz wrote:
> Michael,
> 
> On 9/28/24 13:34, Michael Osipov wrote:
> > On 2024/09/27 15:14:15 Christopher Schultz wrote:
> >> Sebastian,
> >>
> >> On 9/27/24 11:04, Sebastian Trost wrote:
> >>> Francesco,
> >>>
> >>> On 26.09.2024 16:12, Francesco Viscomi wrote:
>  Hi all,
>  I'm not able to understand why I cannot access to
>     http://localhost:8080/manager/html
> 
>  I've configured the user in tomcat.users.xml:
> 
>  
>  
> 
>  I'm using tomcat 9; and jdk17;
> 
>  I've also noted that in my personal pc when try to access manager/html a
>  pop up ask me to login (in my personal pc it works right)
> 
>  While when I try to use it in the company pc it gives me 401
>  unauthorized;
>  I do not know what I have to modify on chrome to get access in manager
>  app,
>  I also use in the company pc Zscaler, but I do not know what I have to
>  change in it (eventually) in order to access the manager app.
> >>> Your corporate browser probably has basic authentication disabled. Check
> >>> this site: https://jigsaw.w3.org/HTTP/Basic
> >>> If there is no basic authentication popup where you can enter username/
> >>> password then this is probably the case.
> >>>
> >>> See: https://answers.microsoft.com/en-us/microsoftedge/forum/all/latest-
> >>> version-of-edge-no-longer-shows-basic/3601252b-e56b-46c0-a088-0f6084eabe47
> >>
> >> I've really had it with Microsoft deciding that HTTP Basic
> >> authentication is just not okay. They seem to have forgotten that TLS
> >> makes it secure.
> > 
> > The reasoning is never to share a long term secret: your password.
> 
> HTTP Digest also requires pre-shared passwords.

There is a subtile difference: the password is never transferred over the wire 
and does not appear on the target server.

> >> HTTP Digest is a nightmare, but they are forcing users onto it.
> > 
> > The key is to use SPNEGO in enterprise environments.
> 
> What about non-enterprise environments?

IMHO, this is irrelevant for Microsoft. In enterprise you do have at least 
SPNEGO or even PKI. For non-enterprise I see only Basic as a viable option.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Hi Chuck,

No - I cannot see any other thread stuck on DB/external resources. The 
application functions as expected, just that I see these threads increasing 
over time. 
The problem was 1st noticed in Tomcat 10.1.26/OpenJRE 17.0.5 and I've tried 
moving to Tomcat 10.1.30/OpenJRE 17.0.12 with no change.

Regards,

-Original Message-
From: Chuck Caldarale  
Sent: Monday, September 30, 2024 1:26 PM
To: Tomcat Users List 
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from n82...@gmail.com. Learn why this is important 
at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

> On Sep 30, 2024, at 12:15, John Williams 
>  wrote:
>
> I had an executor defined before and it had the exact same behavior/problem. 
> Moved to the below model for the connector only after that.

The OP didn't mention the real problem in his original message; it only showed 
up as almost an afterthought in the second posting with the unusable 
attachments:

"Once it reaches close to the maxThreads setting we see connection drops."

I wonder if there's something else blocking the application, such as stuck DB 
operations or other delayed access to an external resource.

  - Chuck

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Chris,



Thanks for your response.



The issue is that the number of threads in the TIMED_WAIT state keeps 
increasing over time. Starts at 150 and keeps growing by 10 every hour. Once it 
reaches close to the maxThreads setting we see connection drops.

This application was working fine with Tomcat 10.0 and JDK 12. Unless the 
application was seeing load, threads would not be created in Tomcat's JVM.



[cid:image003.jpg@01DB1323.CCF9FD00]





You can also see the thread stacks in this image.

[cid:image004.png@01DB1323.CCF9FD00]



Thanks,



-Original Message-
From: Christopher Schultz 
Sent: Monday, September 30, 2024 10:20 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads



[You don't often get email from 
ch...@christopherschultz.net. Learn why 
this is important at https://aka.ms/LearnAboutSenderIdentification ]



CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.





John,



On 9/28/24 05:38, John Williams wrote:

>

> Hi Everyone,

>

> I am running Apache Tomcat 10.1.30 for a web application and notice ~950 
> timed_waiting threads.

>

> The stack trace for these threads is below:

>

> java.base@17.0.12/jdk.internal.misc.Unsafe.park(Native

> e@17.0.12/jdk.internal.misc.Unsafe.park(Native>
>  Method);

> java.base@17.0.12/java.util.concurrent.locks.LockSupport.parkNanos(Loc

> kSupport.java:252) .LockSupport.parkNanos(LockSupport.java:252)>;

> java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchronize

> r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674) lto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchro

> nizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)

> >;

> java.base@17.0.12/java.util.concurrent.LinkedBlockingQueue.poll(Linked

> BlockingQueue.java:460) LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460)>;

> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);

> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);

> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP

> oolExecutor.java:1113);

> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa

> dPoolExecutor.java:1175);

> app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre

> adPoolExecutor.java:659);

> app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta

> skThread.java:63);

> java.base@17.0.12/java.lang.Thread.run(Thread.java:840)

> se@17.0.12/java.lang.Thread.run(Thread.java:840)>;

>

> I am using an executor with maxThreads set to 1500 and minSpareThreads set to 
> 64.

>

> I have not seen this issue with Tomcat 10.0.

>

> What could be the issue and how to resolve it? Any help is much appreciated.



This looks like you have a large number of threads waiting for work.

That seems normal to me if you have a max thread pool of 1500 threads.



What's the problem?



-chris





-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: 
users-h...@tomcat.apache.org

Re: accessing manager app

2024-09-30 Thread Christopher Schultz


Michael,

On 9/28/24 13:34, Michael Osipov wrote:

On 2024/09/27 15:14:15 Christopher Schultz wrote:

Sebastian,

On 9/27/24 11:04, Sebastian Trost wrote:

Francesco,

On 26.09.2024 16:12, Francesco Viscomi wrote:

Hi all,
I'm not able to understand why I cannot access to
   http://localhost:8080/manager/html

I've configured the user in tomcat.users.xml:




I'm using tomcat 9; and jdk17;

I've also noted that in my personal pc when try to access manager/html a
pop up ask me to login (in my personal pc it works right)

While when I try to use it in the company pc it gives me 401
unauthorized;
I do not know what I have to modify on chrome to get access in manager
app,
I also use in the company pc Zscaler, but I do not know what I have to
change in it (eventually) in order to access the manager app.

Your corporate browser probably has basic authentication disabled. Check
this site: https://jigsaw.w3.org/HTTP/Basic
If there is no basic authentication popup where you can enter username/
password then this is probably the case.

See: https://answers.microsoft.com/en-us/microsoftedge/forum/all/latest-
version-of-edge-no-longer-shows-basic/3601252b-e56b-46c0-a088-0f6084eabe47


I've really had it with Microsoft deciding that HTTP Basic
authentication is just not okay. They seem to have forgotten that TLS
makes it secure.


The reasoning is never to share a long term secret: your password.


HTTP Digest also requires pre-shared passwords.


HTTP Digest is a nightmare, but they are forcing users onto it.


The key is to use SPNEGO in enterprise environments.


What about non-enterprise environments?

-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread Christopher Schultz

John,

On 9/30/24 10:30, John Williams wrote:
The issue is that the number of threads in the TIMED_WAIT state keeps 
increasing over time. Starts at 150 and keeps growing by 10 every hour. 
Once it reaches close to the maxThreads setting we see connection drops.

This application was working fine with Tomcat 10.0 and JDK 12. Unless 
the application was seeing load, threads would not be created in 
Tomcat’s JVM.

You can also see the thread stacks in this image.

This list strips out image. Can you post text?

Please post your  configuration taking care to remove any 
secrets.

Thanks,
-chris

-Original Message-
From: Christopher Schultz 
Sent: Monday, September 30, 2024 10:20 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net 
. Learn why this is important at 
https://aka.ms/LearnAboutSenderIdentification  ]

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you recognize the sender and know 
the content is safe.

John,

On 9/28/24 05:38, John Williams wrote:

 >

 > Hi Everyone,

 >

 > I am running Apache Tomcat 10.1.30 for a web application and notice 
~950 timed_waiting threads.

 >

 > The stack trace for these threads is below:

 >

 > java.base@17.0.12/ 
jdk.internal.misc.Unsafe.park(Native

 > e@17.0.12/jdk.internal.misc.Unsafe.park(Native > Method);

 > java.base@17.0.12/ 
java.util.concurrent.locks.LockSupport.parkNanos(Loc 

 > kSupport.java:252) .LockSupport.parkNanos(LockSupport.java:252)>;

 > java.base@17.0.12/ 
java.util.concurrent.locks.AbstractQueuedSynchronize 

 > r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674) lto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchro

 > nizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)

 > >;

 > java.base@17.0.12/ 
java.util.concurrent.LinkedBlockingQueue.poll(Linked 

 > BlockingQueue.java:460) LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460)>;

 > app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);

 > app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);

 > app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP

 > oolExecutor.java:1113);

 > app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa

 > dPoolExecutor.java:1175);

 > app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre

 > adPoolExecutor.java:659);

 > app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta

 > skThread.java:63);

 > java.base@17.0.12/ 
java.lang.Thread.run(Thread.java:840)

 > se@17.0.12/java.lang.Thread.run(Thread.java:840) >;

 >

 > I am using an executor with maxThreads set to 1500 and 
minSpareThreads set to 64.

 >

 > I have not seen this issue with Tomcat 10.0.

 >

 > What could be the issue and how to resolve it? Any help is much 
appreciated.

This looks like you have a large number of threads waiting for work.

That seems normal to me if you have a max thread pool of 1500 threads.

What's the problem?

-chris

-

To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org 

For additional commands, e-mail: users-h...@tomcat.apache.org 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Using HTTP 1.1 over a configured HTTP2 Connector

2024-09-30 Thread Anurag Sharma

Dear Tomcat Team,

I hope this message finds you well.

I am currently facing a challenge regarding the use of HTTP/1.1 for specific 
API endpoints within a servlet configured for HTTP/2. My browser defaults to 
HTTP/2, which complicates the situation as I need to proxy some APIs to a 
server that only supports HTTP/1.1.
Is there a workaround available to enforce HTTP/1.1 for these particular 
endpoints?

Here is out server.xml config. All request from our app is http2 protocol.






  








Thank you so much for your help.



Thanks and Regards,
Anurag Sharma

Re: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread Christopher Schultz


John,

On 9/28/24 05:38, John Williams wrote:


Hi Everyone,

I am running Apache Tomcat 10.1.30 for a web application and notice ~950 
timed_waiting threads.

The stack trace for these threads is below:

java.base@17.0.12/jdk.internal.misc.Unsafe.park(Native
 Method);
java.base@17.0.12/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252);
java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674);
java.base@17.0.12/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1113);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1175);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659);
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63);
java.base@17.0.12/java.lang.Thread.run(Thread.java:840);

I am using an executor with maxThreads set to 1500 and minSpareThreads set to 
64.

I have not seen this issue with Tomcat 10.0.

What could be the issue and how to resolve it? Any help is much appreciated.


This looks like you have a large number of threads waiting for work. 
That seems normal to me if you have a max thread pool of 1500 threads.


What's the problem?

-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread John Williams

Chris,

maxThreads has been set after reviewing the workload. We used a multiple of 64.

The connections are from bots and are unlikely to be pipelined. Connections can 
be bursty - hence, a bigger acceptCount.

Will revert to 10.0 and see. 

Regards,

John

-Original Message-
From: Christopher Schultz  
Sent: Monday, September 30, 2024 5:39 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from ch...@christopherschultz.net. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

John,

On 9/30/24 13:34, John Williams wrote:
> No - I cannot see any other thread stuck on DB/external resources. The 
> application functions as expected, just that I see these threads increasing 
> over time.
> The problem was 1st noticed in Tomcat 10.1.26/OpenJRE 17.0.5 and I've tried 
> moving to Tomcat 10.1.30/OpenJRE 17.0.12 with no change.

If you go back to 10.0 does it still behave as expected?

You have a bit of a weird setup IMHO.

>  maxThreads="1536"

This is a very strange number. Why not 1537?

 >   minSpareThreads="64"

This seems reasonable, but also kind of low given:

>   acceptCount="12000"

So you are willing to accept HUGE numbers of connections and then let them sit 
in the TCP/IP stack's accept queue waiting for a processing thread. Why?

>   connectionTimeout="6"

Ouch. So you will accept huge numbers of connections and then allow them to sit 
there not making any request? This seems like a recipe for disaster.

>   socket.rxBufSize="131072"
 >   socket.txBufSize="131072"

Tweaking the socket buffer sizes is either an indication of an insanely 
well-tuned performance setup or that you don't know what you are doing.
Given the other settings, I'm inclined toward the latter.

 >   maxKeepAliveRequests="1"

Disabling pipelining? Definitely not expecting a high performance service, or 
you are very sure that no client will ever make more than one request.

 >  maxConnections="1536"

You don't want this, especially for NIO. There is no reason to reduce this from 
the default, although your other settings make the non-blocking nature of the 
NIO connector moot (e.g. pipelining is disabled).

Why do you have this collection of odd settings?

-chris

> -Original Message-
> From: Chuck Caldarale 
> Sent: Monday, September 30, 2024 1:26 PM
> To: Tomcat Users List 
> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
>
> [You don't often get email from n82...@gmail.com. Learn why this is 
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.
>
>
>> On Sep 30, 2024, at 12:15, John Williams 
>>  wrote:
>>
>> I had an executor defined before and it had the exact same behavior/problem. 
>> Moved to the below model for the connector only after that.
>
>
> The OP didn't mention the real problem in his original message; it only 
> showed up as almost an afterthought in the second posting with the unusable 
> attachments:
>
> "Once it reaches close to the maxThreads setting we see connection drops."
>
> I wonder if there's something else blocking the application, such as stuck DB 
> operations or other delayed access to an external resource.
>
>- Chuck
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 10.1.30 and Many Timed Waiting threads

2024-09-30 Thread Christopher Schultz

John,

On 9/30/24 13:34, John Williams wrote:

No - I cannot see any other thread stuck on DB/external resources. The 
application functions as expected, just that I see these threads increasing 
over time.
The problem was 1st noticed in Tomcat 10.1.26/OpenJRE 17.0.5 and I've tried 
moving to Tomcat 10.1.30/OpenJRE 17.0.12 with no change.

If you go back to 10.0 does it still behave as expected?

You have a bit of a weird setup IMHO.

>   maxThreads="1536"

This is a very strange number. Why not 1537?

>   minSpareThreads="64"

This seems reasonable, but also kind of low given:

  acceptCount="12000"

So you are willing to accept HUGE numbers of connections and then let 
them sit in the TCP/IP stack's accept queue waiting for a processing 
thread. Why?

  connectionTimeout="6"

Ouch. So you will accept huge numbers of connections and then allow them 
to sit there not making any request? This seems like a recipe for disaster.

  socket.rxBufSize="131072"

>   socket.txBufSize="131072"

Tweaking the socket buffer sizes is either an indication of an insanely 
well-tuned performance setup or that you don't know what you are doing. 
Given the other settings, I'm inclined toward the latter.

>   maxKeepAliveRequests="1"

Disabling pipelining? Definitely not expecting a high performance 
service, or you are very sure that no client will ever make more than 
one request.

>  maxConnections="1536"

You don't want this, especially for NIO. There is no reason to reduce 
this from the default, although your other settings make the 
non-blocking nature of the NIO connector moot (e.g. pipelining is disabled).

Why do you have this collection of odd settings?

-chris

-Original Message-
From: Chuck Caldarale 
Sent: Monday, September 30, 2024 1:26 PM
To: Tomcat Users List 
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads

[You don't often get email from n82...@gmail.com. Learn why this is important 
at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

On Sep 30, 2024, at 12:15, John Williams 
 wrote:

I had an executor defined before and it had the exact same behavior/problem. 
Moved to the below model for the connector only after that.

The OP didn't mention the real problem in his original message; it only showed 
up as almost an afterthought in the second posting with the unusable 
attachments:

"Once it reaches close to the maxThreads setting we see connection drops."

I wonder if there's something else blocking the application, such as stuck DB 
operations or other delayed access to an external resource.

   - Chuck

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Setting Transfer-Encoding: chunked

Re: net::ERR_HTTP2_PROTOCOL_ERROR with 10.1.30

RE: Tomcat 10.1.30 and Many Timed Waiting threads

Re: Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

Re: [OT] accessing manager app

Re: Tomcat 10.1.30 and Many Timed Waiting threads

Re: accessing manager app

RE: Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

Re: accessing manager app

Re: Tomcat 10.1.30 and Many Timed Waiting threads

Using HTTP 1.1 over a configured HTTP2 Connector

Re: Tomcat 10.1.30 and Many Timed Waiting threads

RE: Tomcat 10.1.30 and Many Timed Waiting threads

Re: Tomcat 10.1.30 and Many Timed Waiting threads

16 matches

Site Navigation

Mail list logo

Footer information