Re: Tomcat 9 -> Intermittent 404 (3-4 fails in 20-30 million requests daily sometimes )

[Anurag Kumar]

Thanks, Christopher, for looking into this issue.

Tomcat version:
Server version: Apache Tomcat/9.0.74
Server built: Apr 13, 2023 08:10:39 UTC
Server number: 9.0.74.0


We became aware of this issue a few days ago when it was reported by a
customer due to a critical internal API failure, where the possibility of
unexpected characters was none. Upon investigating the Splunk logs, we
discovered that this issue had been occurring for at least the past 3
months, based on the available three-month log data.

We have a single servlet mapped for all URL patterns, and we log the
requests from this servlet. Internally, we always return a 200 response
code with the appropriate error page and never throw a 404 response.

Here is our Connector configuration:




These 404 issues have been observed on requests created from Chrome,
HttpURLConnection in Java, and AsyncHttpClient in Java. Our servers are
behind an Amazon Load Balancer (ALB), and while ALB operates on HTTP2, our
Tomcat servers are configured for HTTP1.

This issue has been reported on all nine different clusters running the
same Tomcat version. Our test environment closely mirrors the production
environment, but we have been unable to reproduce the issue so far, even
after increasing the number of requests.

It's challenging to identify any specific patterns as the occurrences
appear to be distributed randomly and happen with very simple GET requests.
There was one instance where I was able to reproduce the issue in
production with a straightforward GET request after making 45,000 calls,
but it was never reproduced afterwards through my automation.


Request capture on ALB:-
[image: image.png]


Thanks
Anurag Kumar


On Mon, Oct 16, 2023 at 6:16 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Anurag,
>
> On 10/15/23 04:48, Anurag Kumar wrote:
> >
> > Hi, we are experiencing intermittent 404 errors with both GET and POST
> > calls. These errors are quite rare and have proven difficult to
> > reproduce in our testing environment. However, on our production system,
> > we encounter 3-4 cases daily out of 20-30 million requests where a 404
> > error appears in the Tomcat access logs, and the corresponding call
> > fails to reach the mapped servlet. Interestingly, the same calls work
> > perfectly just a few milliseconds before and after on the same node.
> > This inconsistency is causing significant issues, especially when
> > critical API calls fail and are not automatically retried.
> >
> > Is there any open issue related to this problem that we should be aware
> of?
>
> None that I know of personally.
>
> Can you post your exact Tomcat version, your  configuration
> with any secrets removed and a little more background on the type of
> traffic you are seeing (e.g. HTTP/1.1 v h2, TLS or not, etc.). Are you
> able to tell if these failed requests are part of any kind of pipelined
> requests (HTTP Keep-Alive) or h2 single channels?
>
> Understanding the network topology may be relevant, though its unlikely
> that any lb/rp is doing this, as you can see the logs on the Tomcat
> node. But it may change the way the requests are being handled based
> upon the type of connection between the lb/rp and Tomcat.
>
> Have you double-checked that the URIs are clean and don't contain
> anything unexpected such as lookalike characters, etc.? I suspect this
> is not an issue since you said "critical API calls fail" which leads me
> to understand that you have legitimate customers reporting these
> failures, instead of just investigating unexpected entries in your log
> files.
>
> Is your testing environment reasonably similar to production? What would
> happen if you were to reply a whole day's worth of production-requests
> through your testing environment?
>
> Is there any pattern whatsoever in the failed requests? If you look at
> every failed request for all time, are they randomly distributed
> throughout your URI space, or do you find that some URIs are
> over-represented in your failure data? You may have so few failures that
> you can't draw any conclusions.
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Stale tomcat.pid file prevented Tomcat from starting

[Darryl Baker]

We are running 9.0.78 on RHEL 7. During our monthly patch and reboot cycle one 
the Tomcat running on one system failed to restart. The error said that there 
was a running version of Tomcat with a low PID number. Just rerunning the start 
“systemctl start tomcat” solved the issue. We use the default PID file. I 
looked at the Catalina.sh script and there is a bunch of logic trying to 
prevent this issue but somehow that failed. Is there any know issues with this 
logic?

Darryl Baker

CVE-2023-42794 on 10.1.x

[Donal Anglin]

Hey all,

Sonatype are of the opinion that CVE-2023-42794 is also applicable to the
10.x and 11.x streams of Tomcat and issued the notice:
The Sonatype Security Research team discovered that this vulnerability is
also present and remains unfixed in the 10.x and 11.x branches of Apache
Tomcat.

I assume they are basing that on the 10.1.x branch missing this commit:
https://github.com/apache/tomcat/commit/43b882b8a577684498ab9b8851aa0427216784f7
https://github.com/apache/tomcat/commits/10.1.x/java/org/apache/tomcat/util/http/fileupload/disk/DiskFileItem.java

Are the 10.x and 11.x streams vulnerable to CVE-2023-42794?

Thanks,


*Donal Anglin*

-- 
This message contains proprietary information from Equifax which may be 
confidential. If you are not an intended recipient, please refrain from any 
disclosure, copying, distribution or use of this information and note that 
such actions are prohibited. If you have received this transmission in 
error, please notify by e-mail postmas...@equifax.com 
. Equifax® is a registered trademark of 
Equifax Inc. All rights reserved.

Re: CVE-2023-42794 on 10.1.x

[Mark Thomas]

17 Oct 2023 16:51:38 Donal Anglin :


Hey all,

Sonatype are of the opinion that CVE-2023-42794 is also applicable to 
the

10.x and 11.x streams of Tomcat and issued the notice:
The Sonatype Security Research team discovered that this vulnerability 
is
also present and remains unfixed in the 10.x and 11.x branches of 
Apache

Tomcat.

I assume they are basing that on the 10.1.x branch missing this commit:

https://github.com/apache/tomcat/commit/43b882b8a577684498ab9b8851aa0427216784f7

https://github.com/apache/tomcat/commits/10.1.x/java/org/apache/tomcat/util/http/fileupload/disk/DiskFileItem.java

Are the 10.x and 11.x streams vulnerable to CVE-2023-42794?


Are those versions listed as vulnerable in the announcement for that CVE 
published by the Tomcat project?


Mark




Thanks,


*Donal Anglin*

--
This message contains proprietary information from Equifax which may be
confidential. If you are not an intended recipient, please refrain from 
any
disclosure, copying, distribution or use of this information and note 
that

such actions are prohibited. If you have received this transmission in
error, please notify by e-mail postmas...@equifax.com
. Equifax® is a registered trademark of
Equifax Inc. All rights reserved.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [IE] Re: CVE-2023-42794 on 10.1.x

[Donal Anglin]

No, only 8.x and 9.x.
I assume that Sonatype has done some investigation though.
Do you have any additional context I can share with them to inform their
decision?

*Donal Anglin*

On Tue, Oct 17, 2023 at 6:23 PM Mark Thomas  wrote:

> 17 Oct 2023 16:51:38 Donal Anglin :
>
> > Hey all,
> >
> > Sonatype are of the opinion that CVE-2023-42794 is also applicable to
> > the
> > 10.x and 11.x streams of Tomcat and issued the notice:
> > The Sonatype Security Research team discovered that this vulnerability
> > is
> > also present and remains unfixed in the 10.x and 11.x branches of
> > Apache
> > Tomcat.
> >
> > I assume they are basing that on the 10.1.x branch missing this commit:
> >
> >
> https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-313219e2-454455534531-9e00ea7318970d9b&q=1&e=cff597e0-4029-499f-9554-5de1a3f6fa96&u=https%3A%2F%2Fgithub.com%2Fapache%2Ftomcat%2Fcommit%2F43b882b8a577684498ab9b8851aa0427216784f7
> >
> >
> https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-313219e2-454455534531-f714d7f03a3fde4c&q=1&e=cff597e0-4029-499f-9554-5de1a3f6fa96&u=https%3A%2F%2Fgithub.com%2Fapache%2Ftomcat%2Fcommits%2F10.1.x%2Fjava%2Forg%2Fapache%2Ftomcat%2Futil%2Fhttp%2Ffileupload%2Fdisk%2FDiskFileItem.java
> >
> > Are the 10.x and 11.x streams vulnerable to CVE-2023-42794?
>
> Are those versions listed as vulnerable in the announcement for that CVE
> published by the Tomcat project?
>
> Mark
>
>
> >
> > Thanks,
> >
> >
> > *Donal Anglin*
> >
> > --
> > This message contains proprietary information from Equifax which may be
> > confidential. If you are not an intended recipient, please refrain from
> > any
> > disclosure, copying, distribution or use of this information and note
> > that
> > such actions are prohibited. If you have received this transmission in
> > error, please notify by e-mail postmas...@equifax.com
> > . Equifax® is a registered trademark of
> > Equifax Inc. All rights reserved.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-- 
This message contains proprietary information from Equifax which may be 
confidential. If you are not an intended recipient, please refrain from any 
disclosure, copying, distribution or use of this information and note that 
such actions are prohibited. If you have received this transmission in 
error, please notify by e-mail postmas...@equifax.com 
. Equifax® is a registered trademark of 
Equifax Inc. All rights reserved.

Re: [IE] Re: CVE-2023-42794 on 10.1.x

[Mark Thomas]

17 Oct 2023 18:51:06 Donal Anglin :


No, only 8.x and 9.x.


The question was retorical. I wrote the official announcement.


I assume that Sonatype has done some investigation though.
Do you have any additional context I can share with them to inform 
their

decision?


The onus is on Sonatype to demonstrate that the vulnerability is present 
in one or more Tomcat versions not listed in the official CVE 
announcement.


I'll note that Sonatype have NOT followed the rules of responsible 
disclosure as they have NOT contacted the Tomcat security team of their 
finding.


Mark




*Donal Anglin*

On Tue, Oct 17, 2023 at 6:23 PM Mark Thomas  wrote:


17 Oct 2023 16:51:38 Donal Anglin :


Hey all,

Sonatype are of the opinion that CVE-2023-42794 is also applicable to
the
10.x and 11.x streams of Tomcat and issued the notice:
The Sonatype Security Research team discovered that this 
vulnerability

is
also present and remains unfixed in the 10.x and 11.x branches of
Apache
Tomcat.

I assume they are basing that on the 10.1.x branch missing this 
commit:





https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-313219e2-454455534531-9e00ea7318970d9b&q=1&e=cff597e0-4029-499f-9554-5de1a3f6fa96&u=https%3A%2F%2Fgithub.com%2Fapache%2Ftomcat%2Fcommit%2F43b882b8a577684498ab9b8851aa0427216784f7





https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-313219e2-454455534531-f714d7f03a3fde4c&q=1&e=cff597e0-4029-499f-9554-5de1a3f6fa96&u=https%3A%2F%2Fgithub.com%2Fapache%2Ftomcat%2Fcommits%2F10.1.x%2Fjava%2Forg%2Fapache%2Ftomcat%2Futil%2Fhttp%2Ffileupload%2Fdisk%2FDiskFileItem.java


Are the 10.x and 11.x streams vulnerable to CVE-2023-42794?


Are those versions listed as vulnerable in the announcement for that 
CVE

published by the Tomcat project?

Mark




Thanks,


*Donal Anglin*

--
This message contains proprietary information from Equifax which may 
be
confidential. If you are not an intended recipient, please refrain 
from

any
disclosure, copying, distribution or use of this information and note
that
such actions are prohibited. If you have received this transmission 
in

error, please notify by e-mail postmas...@equifax.com
. Equifax® is a registered trademark 
of

Equifax Inc. All rights reserved.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



--
This message contains proprietary information from Equifax which may be
confidential. If you are not an intended recipient, please refrain from 
any
disclosure, copying, distribution or use of this information and note 
that

such actions are prohibited. If you have received this transmission in
error, please notify by e-mail postmas...@equifax.com
. Equifax® is a registered trademark of
Equifax Inc. All rights reserved.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat minor update

[Aditya Shastri]

Hello,

We have several tomcat instances that use a single CATALINA_HOME which
is a symlink for a specific version. The Tomcat instance we use is
very barebones and doesn't have any of the apps that come with it.

For example,
The CATALINA_HOME points to a symlink
/opt/tomcat/tomcat-9/tomcat-9-latest ->
/opt/tomcat/tomcat-9/apache-tomcat-9.0.80.

Now, if I want to upgrade to apache-tomcat-9.0.82, I normally do the
following steps:
1. Stop all the running instances of tomcat in the various 'CATALINA_BASE'.
2. Update the symlink /opt/tomcat/tomcat-9/tomcat-9-latest from
/opt/tomcat/tomcat-9/apache-tomcat-9.0.80 to
/opt/tomcat/tomcat-9/apache-tomcat-9.0.82.
3. Start all the instances.

This method appears to work, and I read it as the most appropriate method.

My question is, can I change the symlink,
'/opt/tomcat/tomcat-9/tomcat-9-latest', while all the instances are
running and restart the instances when I have downtime?

Does Tomcat load all the CATALINA_HOME jar(s) (not including the
webapps folder) and config to memory thereby not caring if the
libraries have changed or does it realize that something has changed?

Please let me know!
Thanks!

Configuration:
Tomcat version: 9.0.80
OS: Oracle Linux 7

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org