RE: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Eric Robinson]

Mark,

> -Original Message-
> From: Mark Thomas 
> Sent: Thursday, May 30, 2024 9:30 AM
> To: users@tomcat.apache.org
> Subject: Re: Database Connection Requests Initiated but Not Sent on the Wire
> (Some, Not All)
>
> OK.
>
> This is an interim binary patch for 9.0.80 only.
>
> The purpose is to:
> - confirm the proposed change fixes the problem
> - provide you with a workaround in the short term
>
> This is the binary patch:
>
> https://people.apache.org/~markt/dev/classloader-not-found-cache-9.0.80-
> v1.zip
>
> Extract the contents into $CATALINA_HOME/lib
>
> You should end up with:
>
> $CATALINA_HOME/lib/org/apache/...
>
> Usual caveats apply. This is not an official release. Use it at your own 
> risk. Don't
> blame either me or the ASF it is results in alien invasion, a tax bill, the 
> server
> catching fire or anything else unexpected and/or unwanted.
>
> Longer term, I'm not sure this is exactly how I want to fix it in Tomcat. I am
> convinced of the need to cache classes that don't exist but exactly where / 
> how
> to do that and what degree of control the user should have is very much TBD.
>
> I suspect this will be a topic of discussion at Community Over Code at 
> Bratislava
> next week.
>
> I am expecting that any fix won't be in the June release round but should be 
> in
> the July release round.
>
> Let us know how you get on and good luck.
>

The changes have been applied. We'll know at around 9:30 am EST if they have 
had the desired effect. Fingers crossed!


> Mark
>
>
> On 30/05/2024 10:16, Mark Thomas wrote:
> > On 29/05/2024 17:03, Eric Robinson wrote:
> >
> > 
> >
> >> One of the webapps is related to voice reminder messages that go out
> >> to people. The reminders go out sometime after 9 am, which tracks
> >> with the slowdowns.
> >
> > Ack.
> >
> > Something to try while I work on a patch is setting
> > archiveIndexStrategy="bloom" on the resources.
> >
> > You'd configure that in META-INF/context.xml something like this:
> >
> > 
> > 
> >
> > Mark
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

RE: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Eric Robinson]

The results are looking great so far.

Here's what we know:

Before the patch, we had 2 load-balanced tomcats in production for this 
customer. Due to the driver search bottleneck, we were seeing hundreds of stuck 
threads during the slowdown periods. To work around this problem, we threw more 
tomcats at it. With 6 tomcats, the load was spread around enough to keep the 
bottleneck condition from manifesting badly, and users did not complain as 
much. We were still seeing dozens of stuck threads, but not hundreds.

After the patch, we went back to 2 tomcats. During the same timeframe today, 
there have been 1 stuck thread on Tomcat A and 6 on Tomcat B.

If the numbers hold, this works out to roughly a 10,000% improvement.


> -Original Message-
> From: Eric Robinson 
> Sent: Friday, May 31, 2024 5:54 AM
> To: Tomcat Users List 
> Subject: RE: Database Connection Requests Initiated but Not Sent on the Wire
> (Some, Not All)
>
> Mark,
>
> > -Original Message-
> > From: Mark Thomas 
> > Sent: Thursday, May 30, 2024 9:30 AM
> > To: users@tomcat.apache.org
> > Subject: Re: Database Connection Requests Initiated but Not Sent on
> > the Wire (Some, Not All)
> >
> > OK.
> >
> > This is an interim binary patch for 9.0.80 only.
> >
> > The purpose is to:
> > - confirm the proposed change fixes the problem
> > - provide you with a workaround in the short term
> >
> > This is the binary patch:
> >
> > https://people.apache.org/~markt/dev/classloader-not-found-cache-9.0.8
> > 0-
> > v1.zip
> >
> > Extract the contents into $CATALINA_HOME/lib
> >
> > You should end up with:
> >
> > $CATALINA_HOME/lib/org/apache/...
> >
> > Usual caveats apply. This is not an official release. Use it at your
> > own risk. Don't blame either me or the ASF it is results in alien
> > invasion, a tax bill, the server catching fire or anything else unexpected
> and/or unwanted.
> >
> > Longer term, I'm not sure this is exactly how I want to fix it in
> > Tomcat. I am convinced of the need to cache classes that don't exist
> > but exactly where / how to do that and what degree of control the user 
> > should
> have is very much TBD.
> >
> > I suspect this will be a topic of discussion at Community Over Code at
> > Bratislava next week.
> >
> > I am expecting that any fix won't be in the June release round but
> > should be in the July release round.
> >
> > Let us know how you get on and good luck.
> >
>
> The changes have been applied. We'll know at around 9:30 am EST if they have
> had the desired effect. Fingers crossed!
>
>
> > Mark
> >
> >
> > On 30/05/2024 10:16, Mark Thomas wrote:
> > > On 29/05/2024 17:03, Eric Robinson wrote:
> > >
> > > 
> > >
> > >> One of the webapps is related to voice reminder messages that go
> > >> out to people. The reminders go out sometime after 9 am, which
> > >> tracks with the slowdowns.
> > >
> > > Ack.
> > >
> > > Something to try while I work on a patch is setting
> > > archiveIndexStrategy="bloom" on the resources.
> > >
> > > You'd configure that in META-INF/context.xml something like this:
> > >
> > > 
> > > 
> > >
> > > Mark
> > >
> > > 
> > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
>
> Disclaimer : This email and any files transmitted with it are confidential and
> intended solely for intended recipients. If you are not the named addressee 
> you
> should not disseminate, distribute, copy or alter this email. Any views or
> opinions presented in this email are solely those of the author and might not
> represent those of Physician Select Management. Warning: Although Physician
> Select Management has taken reasonable precautions to ensure no viruses are
> present in this email, the company cannot accept responsibility for any loss 
> or
> damage arising from the use of this email or attachments.
> B
> 
> CB  [  X  ܚX KK[XZ[
>
>  \ \  ][  X  ܚX P X ]
>  \X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[
>
>  \ \  Z[ X ]
>  \X K ܙ B
Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

Re: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Mark Thomas]

On 31/05/2024 16:09, Eric Robinson wrote:

The results are looking great so far.


Excellent.


Here's what we know:

Before the patch, we had 2 load-balanced tomcats in production for this 
customer. Due to the driver search bottleneck, we were seeing hundreds of stuck 
threads during the slowdown periods. To work around this problem, we threw more 
tomcats at it. With 6 tomcats, the load was spread around enough to keep the 
bottleneck condition from manifesting badly, and users did not complain as 
much. We were still seeing dozens of stuck threads, but not hundreds.

After the patch, we went back to 2 tomcats.


I appreciate the show of faith! I think that is braver than I would have 
been but it does rather confirm both the problem and the fix.



During the same timeframe today, there have been 1 stuck thread on Tomcat A and 
6 on Tomcat B.


That is great news.


If the numbers hold, this works out to roughly a 10,000% improvement.


Not bad for free support ;)

Seriously, I am glad that we seem to have tracked down the root cause 
and that you have a temporary fix that works until such time (probably 
the July releases) that we can figure out how we want to address caching 
of "not found" classes.


Cheers,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Eric Robinson]

> -Original Message-
> From: Mark Thomas 
> Sent: Friday, May 31, 2024 11:45 AM
> To: users@tomcat.apache.org
> Subject: Re: Database Connection Requests Initiated but Not Sent on the Wire
> (Some, Not All)
>
> On 31/05/2024 16:09, Eric Robinson wrote:
> > The results are looking great so far.
>
> Excellent.
>
> > Here's what we know:
> >
> > Before the patch, we had 2 load-balanced tomcats in production for this
> customer. Due to the driver search bottleneck, we were seeing hundreds of
> stuck threads during the slowdown periods. To work around this problem, we
> threw more tomcats at it. With 6 tomcats, the load was spread around enough
> to keep the bottleneck condition from manifesting badly, and users did not
> complain as much. We were still seeing dozens of stuck threads, but not
> hundreds.
> >
> > After the patch, we went back to 2 tomcats.
>
> I appreciate the show of faith! I think that is braver than I would have been 
> but
> it does rather confirm both the problem and the fix.
>

We verified application functionality during the wee hours and examined the 
logs for evidence of any new issues that may have been introduced by the patch. 
I thought about deploying it to just one server but decided such a test would 
be inconclusive, as we could only know if it really works by allowing it to be 
stress-tested. I had a gut feeling it would be okay. 😊

> > During the same timeframe today, there have been 1 stuck thread on Tomcat
> A and 6 on Tomcat B.
>
> That is great news.
>
> > If the numbers hold, this works out to roughly a 10,000% improvement.
>
> Not bad for free support ;)
>

The best!

> Seriously, I am glad that we seem to have tracked down the root cause and that
> you have a temporary fix that works until such time (probably the July 
> releases)
> that we can figure out how we want to address caching of "not found" classes.
>

We're eager to see how you address it permanently. In the meantime, we're 
delighted because the vendor's recommended "solution" would have been to triple 
the number of tomcat instances, which turns into a maintenance and 
troubleshooting headache. I would bet they have had other cases of slowness 
where the root cause was not isolated. This is a win-win, as it helps both us 
and them.

> Cheers,
>

Cheers back!

> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Christopher Schultz]

Eric,

On 5/31/24 11:09, Eric Robinson wrote:

The results are looking great so far.

Here's what we know:

Before the patch, we had 2 load-balanced tomcats in production for this 
customer. Due to the driver search bottleneck, we were seeing hundreds of stuck 
threads during the slowdown periods. To work around this problem, we threw more 
tomcats at it. With 6 tomcats, the load was spread around enough to keep the 
bottleneck condition from manifesting badly, and users did not complain as 
much. We were still seeing dozens of stuck threads, but not hundreds.

After the patch, we went back to 2 tomcats. During the same timeframe today, 
there have been 1 stuck thread on Tomcat A and 6 on Tomcat B.

If the numbers hold, this works out to roughly a 10,000% improvement.


This deserves a bonus, much of which you should share with markt. ;)

-chris


-Original Message-
From: Eric Robinson 
Sent: Friday, May 31, 2024 5:54 AM
To: Tomcat Users List 
Subject: RE: Database Connection Requests Initiated but Not Sent on the Wire
(Some, Not All)

Mark,


-Original Message-
From: Mark Thomas 
Sent: Thursday, May 30, 2024 9:30 AM
To: users@tomcat.apache.org
Subject: Re: Database Connection Requests Initiated but Not Sent on
the Wire (Some, Not All)

OK.

This is an interim binary patch for 9.0.80 only.

The purpose is to:
- confirm the proposed change fixes the problem
- provide you with a workaround in the short term

This is the binary patch:

https://people.apache.org/~markt/dev/classloader-not-found-cache-9.0.8
0-
v1.zip

Extract the contents into $CATALINA_HOME/lib

You should end up with:

$CATALINA_HOME/lib/org/apache/...

Usual caveats apply. This is not an official release. Use it at your
own risk. Don't blame either me or the ASF it is results in alien
invasion, a tax bill, the server catching fire or anything else unexpected

and/or unwanted.


Longer term, I'm not sure this is exactly how I want to fix it in
Tomcat. I am convinced of the need to cache classes that don't exist
but exactly where / how to do that and what degree of control the user should

have is very much TBD.


I suspect this will be a topic of discussion at Community Over Code at
Bratislava next week.

I am expecting that any fix won't be in the June release round but
should be in the July release round.

Let us know how you get on and good luck.



The changes have been applied. We'll know at around 9:30 am EST if they have
had the desired effect. Fingers crossed!



Mark


On 30/05/2024 10:16, Mark Thomas wrote:

On 29/05/2024 17:03, Eric Robinson wrote:




One of the webapps is related to voice reminder messages that go
out to people. The reminders go out sometime after 9 am, which
tracks with the slowdowns.


Ack.

Something to try while I work on a patch is setting
archiveIndexStrategy="bloom" on the resources.

You'd configure that in META-INF/context.xml something like this:


 

Mark


- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
B

CB  [  X  ܚX KK[XZ[

  \ \  ][  X  ܚX P X ]
  \X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[

  \ \  Z[ X ]
  \X K ܙ B

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Christopher Schultz]

Mark,

On 5/31/24 12:44, Mark Thomas wrote:

On 31/05/2024 16:09, Eric Robinson wrote:

The results are looking great so far.


Excellent.


Here's what we know:

Before the patch, we had 2 load-balanced tomcats in production for 
this customer. Due to the driver search bottleneck, we were seeing 
hundreds of stuck threads during the slowdown periods. To work around 
this problem, we threw more tomcats at it. With 6 tomcats, the load 
was spread around enough to keep the bottleneck condition from 
manifesting badly, and users did not complain as much. We were still 
seeing dozens of stuck threads, but not hundreds.


After the patch, we went back to 2 tomcats.


I appreciate the show of faith! I think that is braver than I would have 
been but it does rather confirm both the problem and the fix.


During the same timeframe today, there have been 1 stuck thread on 
Tomcat A and 6 on Tomcat B.


That is great news.


If the numbers hold, this works out to roughly a 10,000% improvement.


Not bad for free support ;)

Seriously, I am glad that we seem to have tracked down the root cause 
and that you have a temporary fix that works until such time (probably 
the July releases) that we can figure out how we want to address caching 
of "not found" classes.


Yeah... this doesn't seem like a great default policy for a few reasons:

1. Maybe the classes will appear in the future? JSPs? Plugins that 
speculatively-load, then fail, then download/update, then try again? I'm 
grasping at straws a little, here.


2. Huge numbers of cache misses will cause huge numbers of cached "not 
found" entries. Potential DOS? I guess that would be an application bug 
if it's allowing huge numbers of random class-loading requests. Again, 
grasping at straws.


But my spidey-sense it tingling at this one.

Maybe (a) default-off option and (b) limit the total number of cached 
"not found" items to something "smallish" like 100? 1000? 1? Just 
enough to not fill the heap if something goes terribly wrong.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Eric Robinson]

> -Original Message-
> From: Christopher Schultz 
> Sent: Friday, May 31, 2024 12:38 PM
> To: users@tomcat.apache.org
> Subject: Re: Database Connection Requests Initiated but Not Sent on the Wire
> (Some, Not All)
>
> Mark,
>
> On 5/31/24 12:44, Mark Thomas wrote:
> > On 31/05/2024 16:09, Eric Robinson wrote:
> >> The results are looking great so far.
> >
> > Excellent.
> >
> >> Here's what we know:
> >>
> >> Before the patch, we had 2 load-balanced tomcats in production for
> >> this customer. Due to the driver search bottleneck, we were seeing
> >> hundreds of stuck threads during the slowdown periods. To work around
> >> this problem, we threw more tomcats at it. With 6 tomcats, the load
> >> was spread around enough to keep the bottleneck condition from
> >> manifesting badly, and users did not complain as much. We were still
> >> seeing dozens of stuck threads, but not hundreds.
> >>
> >> After the patch, we went back to 2 tomcats.
> >
> > I appreciate the show of faith! I think that is braver than I would
> > have been but it does rather confirm both the problem and the fix.
> >
> >> During the same timeframe today, there have been 1 stuck thread on
> >> Tomcat A and 6 on Tomcat B.
> >
> > That is great news.
> >
> >> If the numbers hold, this works out to roughly a 10,000% improvement.
> >
> > Not bad for free support ;)
> >
> > Seriously, I am glad that we seem to have tracked down the root cause
> > and that you have a temporary fix that works until such time (probably
> > the July releases) that we can figure out how we want to address
> > caching of "not found" classes.
>
> Yeah... this doesn't seem like a great default policy for a few reasons:
>
> 1. Maybe the classes will appear in the future? JSPs? Plugins that 
> speculatively-
> load, then fail, then download/update, then try again? I'm grasping at straws 
> a
> little, here.
>
> 2. Huge numbers of cache misses will cause huge numbers of cached "not
> found" entries. Potential DOS? I guess that would be an application bug if 
> it's
> allowing huge numbers of random class-loading requests. Again, grasping at
> straws.
>

Would it, though? I don't know what a negative cache entry would look like, but 
it seems to me that it would not have to create duplicates.

> But my spidey-sense it tingling at this one.
>
> Maybe (a) default-off option and (b) limit the total number of cached "not
> found" items to something "smallish" like 100? 1000? 1? Just enough to not
> fill the heap if something goes terribly wrong.
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Christopher Schultz]

Eric,

On 5/31/24 13:44, Eric Robinson wrote:

-Original Message-
From: Christopher Schultz 
Sent: Friday, May 31, 2024 12:38 PM
To: users@tomcat.apache.org
Subject: Re: Database Connection Requests Initiated but Not Sent on the Wire
(Some, Not All)

Mark,

On 5/31/24 12:44, Mark Thomas wrote:

On 31/05/2024 16:09, Eric Robinson wrote:

The results are looking great so far.


Excellent.


Here's what we know:

Before the patch, we had 2 load-balanced tomcats in production for
this customer. Due to the driver search bottleneck, we were seeing
hundreds of stuck threads during the slowdown periods. To work around
this problem, we threw more tomcats at it. With 6 tomcats, the load
was spread around enough to keep the bottleneck condition from
manifesting badly, and users did not complain as much. We were still
seeing dozens of stuck threads, but not hundreds.

After the patch, we went back to 2 tomcats.


I appreciate the show of faith! I think that is braver than I would
have been but it does rather confirm both the problem and the fix.


During the same timeframe today, there have been 1 stuck thread on
Tomcat A and 6 on Tomcat B.


That is great news.


If the numbers hold, this works out to roughly a 10,000% improvement.


Not bad for free support ;)

Seriously, I am glad that we seem to have tracked down the root cause
and that you have a temporary fix that works until such time (probably
the July releases) that we can figure out how we want to address
caching of "not found" classes.


Yeah... this doesn't seem like a great default policy for a few reasons:

1. Maybe the classes will appear in the future? JSPs? Plugins that 
speculatively-
load, then fail, then download/update, then try again? I'm grasping at straws a
little, here.

2. Huge numbers of cache misses will cause huge numbers of cached "not
found" entries. Potential DOS? I guess that would be an application bug if it's
allowing huge numbers of random class-loading requests. Again, grasping at
straws.



Would it, though? I don't know what a negative cache entry would look like, but 
it seems to me that it would not have to create duplicates.


I was thinking of cache entries for large numbers of different classes, 
all of which were "not found". Not one class being requested over and 
over again. It's just lots of String keys in a hash map or whatever. Not 
horrific, but can get out of control of Something Goes Wrong.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Need help installing SSL certificate in tomcat keystore

[Christopher Schultz]

Mark,

On 5/30/24 08:46, Fung-A-Fat, Mark wrote:
I am running a java web app on windows 2019 server and need some help 
getting the SSL certificate installed into my keystore.


I am running tomcat 9.x and java 11

I am able to generate a certificate request using both keytool and/or 
openssl


For both the CSR file looks like this, but the openssl also generates a 
private key xxx.


-BEGIN NEW CERTIFICATE REQUEST-

MIIC2TCCAcECAQAwZDELMAkGA1UEBhMCdXMxCzAJBgNVBAgTAm1hMRAwDgYDVQQH

-END NEW CERTIFICATE REQUEST-

Private key from OPENSSL

-BEGIN PRIVATE KEY-
MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQC5EqmuGM9nRQ5n
-END PRIVATE KEY-


You may have compromised your private key by posting it like this. I 
would start everything over again from scratch, starting with generating 
a new private key and CSR.


I use the CSR to submit a request to my company’s certificate server and 
I am able to download 2 files in DER format


The downloaded certificate has a name certnew.cer, the downloaded chain 
certificate has a name cernew.p7b and both appear to be binary because 
when I open them in notepad++ they are unreadable


.p12 and .p7 files are always binary. Are you able to get the files as 
PEM? That is, IMHO, the most convenient package format.


Not sure how I go about importing converting and importing these into my 
keystore using keytool.


The documenation is confusing to me as to what needs to be done.

https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html 
the section on 
importing the certificate does nto go into how to convert or merge the 
certificate or the certificate chain and also does not say anyting about 
a private keyfile


Has anyone out there done this consistenly and successfully.


You should be able to use keytool -importcert as described here:

https://stackoverflow.com/questions/15814569/import-pkcs7-chained-certificate-using-keytool-command-to-jks

When you do all of this start-to-finish, basically you do the following:

1. $ keytool -genkeypair -alias 'mykey' (creates key + self-signed cert 
in keystore, plus CSR)


2. Send CSR to CA for signing, get signed cert in return

3. $ keytool -importcert -alias 'mykey'

This will UPDATE THE CERT in your keystore with the one signed by the 
CA. Now, you are ready to use the signed certificate with Tomcat.


But definitely start over with a new private key. The one you posted 
shouldn't be trusted anymore.


Hope that helps,
-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Terence M. Bandoian]

On 5/31/2024 11:44 AM, Mark Thomas wrote:

On 31/05/2024 16:09, Eric Robinson wrote:

The results are looking great so far.


Excellent.


Here's what we know:

Before the patch, we had 2 load-balanced tomcats in production for 
this customer. Due to the driver search bottleneck, we were seeing 
hundreds of stuck threads during the slowdown periods. To work around 
this problem, we threw more tomcats at it. With 6 tomcats, the load 
was spread around enough to keep the bottleneck condition from 
manifesting badly, and users did not complain as much. We were still 
seeing dozens of stuck threads, but not hundreds.


After the patch, we went back to 2 tomcats.


I appreciate the show of faith! I think that is braver than I would 
have been but it does rather confirm both the problem and the fix.


During the same timeframe today, there have been 1 stuck thread on 
Tomcat A and 6 on Tomcat B.


That is great news.


If the numbers hold, this works out to roughly a 10,000% improvement.


Not bad for free support ;)

Seriously, I am glad that we seem to have tracked down the root cause 
and that you have a temporary fix that works until such time (probably 
the July releases) that we can figure out how we want to address 
caching of "not found" classes.


Cheers,

Mark


Kudos!

Would this mean that a "not found" class would be forever "not found" 
until a restart? How does Tomcat handle jars or classes added or removed 
while it's running now? Could a class be removed from the "not found" 
cache if a change was detected? Would it be worth maintaining a list of 
all available classes?


Really well done!

-Terence

RE: Database Connection Requests Initiated but Not Sent on the Wire (Some, Not All)

[Eric Robinson]

Chris,

> -Original Message-
> From: Christopher Schultz 
> Sent: Friday, May 31, 2024 12:50 PM
> To: users@tomcat.apache.org
> Subject: Re: Database Connection Requests Initiated but Not Sent on the Wire
> (Some, Not All)
>
> Eric,
>
> On 5/31/24 13:44, Eric Robinson wrote:
> >> -Original Message-
> >> From: Christopher Schultz 
> >> Sent: Friday, May 31, 2024 12:38 PM
> >> To: users@tomcat.apache.org
> >> Subject: Re: Database Connection Requests Initiated but Not Sent on
> >> the Wire (Some, Not All)
> >>
> >> Mark,
> >>
> >> On 5/31/24 12:44, Mark Thomas wrote:
> >>> On 31/05/2024 16:09, Eric Robinson wrote:
>  The results are looking great so far.
> >>>
> >>> Excellent.
> >>>
>  Here's what we know:
> 
>  Before the patch, we had 2 load-balanced tomcats in production for
>  this customer. Due to the driver search bottleneck, we were seeing
>  hundreds of stuck threads during the slowdown periods. To work
>  around this problem, we threw more tomcats at it. With 6 tomcats,
>  the load was spread around enough to keep the bottleneck condition
>  from manifesting badly, and users did not complain as much. We were
>  still seeing dozens of stuck threads, but not hundreds.
> 
>  After the patch, we went back to 2 tomcats.
> >>>
> >>> I appreciate the show of faith! I think that is braver than I would
> >>> have been but it does rather confirm both the problem and the fix.
> >>>
>  During the same timeframe today, there have been 1 stuck thread on
>  Tomcat A and 6 on Tomcat B.
> >>>
> >>> That is great news.
> >>>
>  If the numbers hold, this works out to roughly a 10,000% improvement.
> >>>
> >>> Not bad for free support ;)
> >>>
> >>> Seriously, I am glad that we seem to have tracked down the root
> >>> cause and that you have a temporary fix that works until such time
> >>> (probably the July releases) that we can figure out how we want to
> >>> address caching of "not found" classes.
> >>
> >> Yeah... this doesn't seem like a great default policy for a few reasons:
> >>
> >> 1. Maybe the classes will appear in the future? JSPs? Plugins that
> >> speculatively- load, then fail, then download/update, then try again?
> >> I'm grasping at straws a little, here.
> >>
> >> 2. Huge numbers of cache misses will cause huge numbers of cached
> >> "not found" entries. Potential DOS? I guess that would be an
> >> application bug if it's allowing huge numbers of random class-loading
> >> requests. Again, grasping at straws.
> >>
> >
> > Would it, though? I don't know what a negative cache entry would look like,
> but it seems to me that it would not have to create duplicates.
>
> I was thinking of cache entries for large numbers of different classes, all of
> which were "not found". Not one class being requested over and over again. 
> It's
> just lots of String keys in a hash map or whatever. Not horrific, but can get 
> out
> of control of Something Goes Wrong.
>

Gotcha. In that case, if there were hundreds of different classes not found, 
then it seems the app must have bigger problems, like entirely missing lib 
folders or something, and the system would barely function, if at all.  I 
realize I'm speaking largely from ignorance here.

> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 9 memory leak message

[Ying Jin]

We removed the ojdbc8 driver jar from web-inf/lib from the web application
and kept the ojdbc8 jar file in the Tomcat/lib folder, but we still can see
the following memory link warning message whenever we redeploy the web
application. We use the Tomcat 9 server in the Linux environment.

The other warning message is about the "validateFile Problem with jar file
/tomcat/lib/jolokia.jar. My question is if we can safely ignore these
warning messages or not.

It would be great if you can shed some light on this issue.
[image: image.png]


Your help is much appreciated!
Jenny

Re: Tomcat 9 memory leak message

[Christopher Schultz]

Jenny,

On 5/31/24 14:52, Ying Jin wrote:
We removed the ojdbc8 driver jar from web-inf/lib from the web 
application and kept the ojdbc8 jar file in the Tomcat/lib folder, but 
we still can see the following memory link warning message whenever we 
redeploy the web application. We use the Tomcat 9 server in the Linux 
environment.


This list strips attachments. Can you re-post with text-only?

The other warning message is about the "validateFile Problem with jar 
file /tomcat/lib/jolokia.jar. My question is if we can safely ignore 
these warning messages or not.


It would be great if you can shed some light on this issue.


If the message is something like "driver cannot be unloaded" then check 
Tomcat with a debugger or even something like JVisualVM to see how many 
WebappClassLoaders you have in memory.


If the driver causes the web application ClassLoader to be "pinned" in 
memory, then it will never be removed and all those classes will 
continue to use-up heap space until you restart the JVM. This gets worse 
every time you reload your application without restarting the JVM. The 
Manager application web UI can help you diagnose these a little.


The validation problem with the Jolokia JAR file will depend upon 
exactly what it says. I would first get a replacement copy of the 
Jolokia JAR file before bothering to try to diagnose it any further.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9 memory leak message

[Ying Jin]

Chris,

Thanks for your reply!

We already removed the ojdbc8.jar file from the application's Web-inf/lib
folder as suggested in the following post, however, we still got the
warning messages below after the application is deployed to the Tomcat 9
server.

https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-put-in-tomcat-home-lib-folder

WARNING: The web application [Our Web Application Name] appears to have
started a thread named [InterruptTimer] but has failed to stop it. This is
very likely to create a memory leak. Stack trace of thread:

WARNING: The web application [Our Web Application Name] appears to have
started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but has
failed to stop it. This is very likely to create a memory leak. Stack trace
of thread:

I also read some posts saying these warning messages can be safely ignored
if the Tomcat version is greater than 7.0. I'm not sure if this is correct
or not.

Please advise,

Many thanks!
Jenny

On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Jenny,
>
> On 5/31/24 14:52, Ying Jin wrote:
> > We removed the ojdbc8 driver jar from web-inf/lib from the web
> > application and kept the ojdbc8 jar file in the Tomcat/lib folder, but
> > we still can see the following memory link warning message whenever we
> > redeploy the web application. We use the Tomcat 9 server in the Linux
> > environment.
>
> This list strips attachments. Can you re-post with text-only?
>
> > The other warning message is about the "validateFile Problem with jar
> > file /tomcat/lib/jolokia.jar. My question is if we can safely ignore
> > these warning messages or not.
> >
> > It would be great if you can shed some light on this issue.
>
> If the message is something like "driver cannot be unloaded" then check
> Tomcat with a debugger or even something like JVisualVM to see how many
> WebappClassLoaders you have in memory.
>
> If the driver causes the web application ClassLoader to be "pinned" in
> memory, then it will never be removed and all those classes will
> continue to use-up heap space until you restart the JVM. This gets worse
> every time you reload your application without restarting the JVM. The
> Manager application web UI can help you diagnose these a little.
>
> The validation problem with the Jolokia JAR file will depend upon
> exactly what it says. I would first get a replacement copy of the
> Jolokia JAR file before bothering to try to diagnose it any further.
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>