Re: tomcat 10.1.33 random rare 500 response status for http2 upgrade with tls

2025-02-28 Thread Cenk Pekyaman
On Wed, Feb 26, 2025 at 5:31 PM Mark Thomas  wrote:

> On 26/02/2025 12:04, Mark Thomas wrote:
> > On 26/02/2025 08:16, Mark Thomas wrote:
> >> On 13/02/2025 10:04, Rémy Maucherat wrote:
> >>> On Thu, Feb 13, 2025 at 9:41 AM Cenk Pekyaman
> >>>  wrote:
> 
>  We run tomcat on java17 with the embedded tomcat setup.
>  We have http and https connectors and we have http2 upgradeProtocol
> for
>  both.
> 
>  We recently upgraded from 9.0.88 to 10.1.24 to work on javax to
> jakarta
>  migration, and after a while, upgraded to 10.1.33.
>  After the upgrade, we started to see random and rare 500 errors for
>  some of
>  the http2 GET requests over https.
>  When reproducing the error on our development machines, we could see
>  the
>  following trace on the server:
>  ```
>    java.io.IOException: null
> >>>
> >>> Tomcat now sets an IO exception to trigger ReadListener.onError with
> >>> an appropriate error in that case. So things seem normal so far.
> >>
> >> Following up on this.
> >>
> >> Tomcat is behaving as if the client has reset the stream before the
> >> client sent all of the data. If that is what the client is doing then
> >> this behaviour is expected. However, that then raises the question why
> >> is the reset being sent. If the client isn't resetting the stream then
> >> there is definitely a Tomcat bug here. We need to look at the test
> >> case you've provided. I'm planning on starting that today.
> >
> > Thanks again for the test case. It makes debugging issues so much easier
> > when the report includes a reproducible test case.
> >
> > I am able to reproduce this issue with the latest 12.0.x code.
> >
> > It does look like there is a Tomcat bug here. I am currently working on
> > tracking down the root cause.
>
> Found it and fixed the issue. The fix will be in the March releases.
>
> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
Thanks for looking into this.
Glad to hear the test case helped and you were able to find the issue.

Cenk.


Re: Handling CloseNowException in Tomcat 9.0

2025-02-28 Thread Mark Thomas

On 27/02/2025 19:56, Banana Kanana wrote:

Hi,

We are using Apache Tomcat 9.0 and frequently see logs related to
CloseNowException in one of our applications. This exception occurs on
multiple operating systems, including OpenSUSE, Ubuntu, and Windows, and in
different parts of our codebase.

 From reading the Javadoc, our understanding is that this exception signals
that a connection should be closed. We are unsure about the best way to
handle it in our application.

• Should we rethrow it so that Tomcat can handle it properly?


That isn't necessary but it may help Tomcat end the request/response faster.


• What would be the implications if we just catch and log the exception
without rethrowing it? Could this interfere with Tomcat’s expected behavior
in any way?


It shouldn't. Tomcat should mark the request/response as having an error 
so the request/response should be closed once control returns to Tomcat 
whether the applications re-throws the exception or not.


My recommendation when an application sees this error would be:
- log it (if the application wants to)
- stop processing the current request/response as quickly as possible
- clean-up any resources used by the current request/response
- rethrow the exception (may enable Tomcat to skip some processing)

How much impact rethrowing has is going to depend on circumstances. I 
think it will only help but you might want to test that.


Mark




We observe multiple variations of stack traces, but here is one example,
truncated to omit internal details. Please note that the log entry is
generated by our class, not by Tomcat:

2025-02-27 00:04:44.740 ERROR [ErrorFilter] [https-jsse-nio-443-exec-3604]
[] [] Request by anonymous failed for GET https://www.nnn.
org.apache.coyote.CloseNowException: Connection [1775817], Stream [27],
This stream is in state [CLOSED_RST_RX] and is not writable at
org.apache.coyote.http2.Stream.doStreamCancel(Stream.java:293) at
org.apache.coyote.http2.Http2UpgradeHandler.reserveWindowSize(Http2UpgradeHandler.java:941)
at
org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1050)
at
org.apache.coyote.http2.Stream$StreamOutputBuffer.doWrite(Stream.java:965)
at
org.apache.coyote.http2.Http2OutputBuffer.doWrite(Http2OutputBuffer.java:57)
at org.apache.coyote.Response.doWrite(Response.java:608) at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:333)
at
org.apache.catalina.connector.OutputBuffer.appendByteArray(OutputBuffer.java:750)
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:671)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:380)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:358)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:103)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:95)

For additional context:

Tomcat versions: multiple versions. The provided stack traces is from
9.0.87 but we see this in 9.0.98 as well

• Operating systems affected: OpenSUSE, Ubuntu, Windows

• HTTP/2 enabled: Yes, in the environments where this exception occurs


We have reviewed the following resources without finding a clear answer:

• Tomcat’s official documentation

• Previous discussions in the user mailing list archives

• Related posts on Stack Overflow


We would appreciate guidance on the best approach to handling
CloseNowException in our application to ensure compatibility with Tomcat.


Thanks in advance for your help.
Daniel




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org