NoClassDef error

[Mcalexander, Jon J.]

Hello,
I have an application team that is receiving the following exception after 
upgrading to Tomcat 9.0.90.

Exception java.lang.NoClassDefFoundError: 
org/apache/tomcat/dbcp/pool2/BasePooledObjectFactory

If they revert back to 9.0.89 it once again works. Was there a change to this 
component between 89 and 90?

Thank you,

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.

Re: NoClassDef error

[Mark Thomas]

On 22/07/2024 15:31, Mcalexander, Jon J. wrote:

Hello,
I have an application team that is receiving the following exception after 
upgrading to Tomcat 9.0.90.

Exception java.lang.NoClassDefFoundError: 
org/apache/tomcat/dbcp/pool2/BasePooledObjectFactory

If they revert back to 9.0.89 it once again works. Was there a change to this 
component between 89 and 90?


Unused code was removed. That was one of the classes that was removed.

Applications should not be using Tomcat's internal fork of Commons Pool. 
They should add the Commons Pool JAR to WEB-INF/lib and use it directly.


Mark



Thank you,

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: NoClassDef error

[Mcalexander, Jon J.]

Thank you Mark!

From: Mark Thomas 
Sent: Monday, July 22, 2024 9:59 AM
To: users@tomcat.apache.org
Subject: Re: NoClassDef error

On 22/07/2024 15: 31, Mcalexander, Jon J. wrote: > Hello, > I have an 
application team that is receiving the following exception after upgrading to 
Tomcat 9. 0. 90. > > Exception java. lang. NoClassDefFoundError: 
org/apache/tomcat/dbcp/pool2/BasePooledObjectFactory


On 22/07/2024 15:31, Mcalexander, Jon J. wrote:

> Hello,

> I have an application team that is receiving the following exception after 
> upgrading to Tomcat 9.0.90.

>

> Exception java.lang.NoClassDefFoundError: 
> org/apache/tomcat/dbcp/pool2/BasePooledObjectFactory

>

> If they revert back to 9.0.89 it once again works. Was there a change to this 
> component between 89 and 90?



.Unused code was removed. That was one of the classes that was removed.



Applications should not be using Tomcat's internal fork of Commons Pool.

They should add the Commons Pool JAR to WEB-INF/lib and use it directly.



Mark



>

> Thank you,

>

> Dream * Excel * Explore * Inspire

> Jon McAlexander

> Senior Infrastructure Engineer

> Asst. Vice President

> He/His

>

> Middleware Product Engineering

> Enterprise CIO | EAS | Middleware | Infrastructure Solutions

>

> 8080 Cobblestone Rd | Urbandale, IA 50322

> MAC: F4469-010

> Tel 515-988-2508 | Cell 515-988-2508

>

> jonmcalexan...@wellsfargo.com>

> This message may contain confidential and/or privileged information. If you 
> are not the addressee or authorized to receive this for the addressee, you 
> must not use, copy, disclose, or take any action based on this message or any 
> information herein. If you have received this message in error, please advise 
> the sender immediately by reply e-mail and delete this message. Thank you for 
> your cooperation.

>

>



-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: 
users-h...@tomcat.apache.org

Re: Reg: tomcat CPU spikes

[Christopher Schultz]

Jalaj,

On 7/19/24 14:28, Jalaj Asher wrote:

This is the warning message we get when cachingAllowed is not set to false

org.apache.catalina.webresources.Cache.getResource Unable to add the resource 
at [/WEB-INF/classes/] to the cache for web application [/x] because there 
was insufficient free space available after evicting expired cache entries - 
consider increasing the maximum size of the cache.


Okay, I see it. Specifically, it is a WARN message which is usually not 
suppressed in a production configuration.


@markt @remm what do you think about making this another of those "WARN 
the first time, DEBUG thereafter" kinds of messages?


It seems like if a cache is full, the operator SHOULD get a 
notification, but if the cache is thrashing, printing an error a huge 
number of times doesn't seem like it's terribly helpful. It just fills 
the disk.


-chris


-Original Message-
From: Jalaj Asher
Sent: Tuesday, July 16, 2024 1:30 PM
To: Tomcat Users List 
Subject: RE: Reg: tomcat CPU spikes


space". Which was very quickly filling up our disk space as well as
increasing disk IO causing latency concerns.

1. Also interesting. Can you post one of those messages here? Was there a stack 
trace shown or just the warning?

 It is just the warning. No stack trace. I will work on recreating this 
since all our environments has this disabled.

2. Interesting. How much static content do you have? This seems like a good 
use-case for a reverse-proxy to handle your staticcontent for you.
We have not collated the complete size of it. But are reasons we cannot do that.

Also I was reviewing some older heap dumps and I could see that the jars are 
getting cached in tomcat even with cachingAllowed=false.

Also this is not a consistent issue once it happens it takes sometime for the 
stack to go away as well as post tomcat reboots the problem goes away with the 
same settings and we do see that the wars are getting deployed during tomcat 
startup as well.

Regards

Jalaj P Asher


-Original Message-
From: Christopher Schultz 
Sent: Tuesday, July 16, 2024 10:05 AM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source outside of 
eClinicalWorks. Always use caution when opening attachments, clicking links, or 
when responding to this email. If you feel this is a phishing scam, please use 
the Phish Alert Report button in Outlook.


Jalaj,

On 7/15/24 18:18, Jalaj Asher wrote:

We ran into 2 issues
1. We needed to allocate significant amount of -XMX  for heap space,
if we allowed caching, since increasing memory by a few hundred MB as
well was not enough.

Interesting. How much static content do you have? This seems like a good 
use-case for a reverse-proxy to handle your static content for you.


2. Also with the setting being  enabled, it generated logs stating
"could not add a resource  as there wasn’t enough
space". Which was very quickly filling up our disk space as well as
increasing disk IO causing latency concerns.

Also interesting. Can you post one of those messages here? Was there a stack 
trace shown or just the warning?

-chris


-Original Message-
From: Christopher Schultz 
Sent: Monday, July 15, 2024 4:19 PM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source outside of 
eClinicalWorks. Always use caution when opening attachments, clicking links, or 
when responding to this email. If you feel this is a phishing scam, please use 
the Phish Alert Report button in Outlook.


Jalaj,

On 7/15/24 15:03, Jalaj Asher wrote:

Yeah I was wondering the same as this has been in place since a few
years now atleast 4 years since cachingAllowed had some changes in
tomcat 8 which was resulting in it caching all static content as well
as jsps and jars and our though process was if we have static content
being cached on the client end and jsps in the work folder each time
on access we don’t need the cache.

Does the cache actively hurt you?


Is there a way to cache just the jars and not every thing else in memory ?


I think the short answer is "no there is not a way to do this" but I may be 
wrong.

The long answer might be "maybe, but you will have to play games with  
and  and maybe some other things to get it working.

I would save yourself some complexity and simply enable caching.

-chris


-Original Message-
From: Christopher Schultz 
Sent: Friday, July 12, 2024 4:02 PM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source outside of 
eClinicalWorks. Always use caution when opening attachments, clicking links, or 
when responding to this email. If you feel this is a phishing scam, please use 
the Phish Alert Report button in Outlook.


Jalaj,

On 7/12/24 10:19, Jalaj Asher wrote:

Thank you Chuck and John for the responses.

Just a few points from the t

Re: Intermittent Missing Content-Type

[Christopher Schultz]

Simon,

On 7/18/24 14:54, Simon Arame wrote:

Greetings folks,

According to JavaServer Pages™ specs v2.3 #JSP4.2, when a JSP page does not
provide the TYPE value of the contentType attribute, "the initial content
type is “text/html” for JSP pages in standard syntax"

With our relatively big web application, we are experiencing an
intermittent issue which is a missing Content-Type header on some JSP pages
and the issue goes away when we use 9.0.90+ or set
org.apache.catalina.connector.RECYCLE_FACADES to true.


This is likely because the default for that setting changed in 9.0.90 
from "false" to "true":


https://tomcat.apache.org/tomcat-9.0-doc/changelog.html#Tomcat_9.0.90_(remm)

If the issue goes away when discarding facades, my first response would 
be to suggest that there is an issue in the application which is 
retaining a reference to a response object and damaging that content-type.


It's also possible that there is a subtle bug in Tomcat where the 
content-type isn't being reset as expected.



Some of our clients are not easily updating their tomcat version so I would
like to know what we are doing wrong maybe in our custom Filter that causes
the occasional missing content-type when facades are not recycled.


It may be very difficult to track-down the source of this issue. If you 
are able to provide a simple JSP that behaves in this way and perhaps a 
Filter that sometimes causes an issue, we may be able to help track it 
down. But if you can't share code, we can't help.


It may be better for you to simply reconfigure your clients to disable 
facade reuse either by using the RECYCLE_FACADES=true system property or 
to use the discardFacades="true"  attribute which has been in 
Tomcat 9.0.x since 9.0.31:


https://tomcat.apache.org/tomcat-9.0-doc/changelog.html#Tomcat_9.0.31_(markt)


I got help from Konstantin Kolinko to locate some historical tomcat code
changes. What I was trying to determine is details about the "recycle"
method of the Request implementation and with Konstantin's help (see
below), I make the reconciliation with the CoyoteRequest and the
CoyoteRequestFacade classes. Now I can see the commit 301781 simply adds
those. Is this coming straight on from a CVS repository ? If yes, can the
CVS repository archive can still be publicly accessed ? At this point, I am
not sure if commit 302827 (
https://svn.apache.org/viewvc?view=revision&revision=302827 ) would explain
what i was searching for. My point is that at 301781 (
https://svn.apache.org/viewvc?view=revision&revision=301781 ), the facade
is only cleared if Constants.SECURITY is true and now since 9.0.90, facades
are cleared by default unless you change the option
org.apache.catalina.connector.RECYCLE_FACADES to false.


The facade is always "cleared" in one way or the other. When the facade 
will be re-used (aka "recycled"), the information that had previously 
been in the facade needs to be "cleared" by nulling-out references and 
setting primitive fields back to their defaults.


If the facade will be discarded and a new object created in its place, 
there is no reason to explicitly "clear" the old object... Tomcat simply 
lets the reference go out of scope and the GC will clean it up at some 
point. So when the facade will be GC'd, we don't bother blanking-out all 
those fields.



So following that logic, I found out what facade recycling was but I don't
know why we rely on it.


This is a performance optimization that IMHO is mostly historical at 
this point. Requests ought to be fast enough that the cycle of request 
-> new object -> end request -> GC should be very cheap as most requests 
will not "survive" a minor GC action. The effects of "new/malloc" and 
(conceptually) delete/free are essentially zero with modern Java memory 
managers for short-lived objects.



The missing content-type occurs when we navigate quickly between jsp pages.

Here are some of the actions we performed in our custom Filter,
1. we are conditionally setting some headers on the Response object.


Where does this happen? In the doFilter method, or elsewhere? If you are 
not retaining any references to the response (or request, for that 
matter) after doFilter ends, then this issue shouldn't be "your fault". 
But if you are stashing references to the HttpServletRequest or 
HttpServletResponse into the request, response, session, some 
object-level or class-level member, etc. then you might be causing this 
problem yourself.



2. we are conditionally calling the RequestDispatcher.include method to add
a JSP head ( stuffs like  )


This sounds like a job for a view-framework, but should have nothing to 
do with the behavior you are describing.



3. depending on an external authentication call result we are either
sending a redirect or calling FilterChain.doFilter


Sounds okay to me so far.


Please note that we tried the following without success:
- declaring a default-content-type inside a jsp-property-group as
prescribed by the JavaServer