Which is better: many concurrent requests or a deep backlog?

[mwood]

For various reasons I have Tomcat behind Apache HTTPD via
mod_proxy_ajp, and I'm looking for good ways to think about the proxy
connection pool.  I'm wondering if I should let HTTPD make as many
connections as it wants, or set a maximum pool size and let requests
wait in the proxy.  This is probably a complex issue and I want to be
sure I'm considering the right aspects.

I think I do want persistent proxy connections.  This seems to work
well and it saves setup/teardown work.  So I have
"ProxyPass...enablereuse=on" (which I think is default, but I hate
relying on defaults for things I care about).

Tomcat v9.0.90, HTTPD 2.4.62.  There are no obvious problems at the
moment.  I just want to be sure I'm doing all I can to keep this
back-end service from ever being a bottleneck for a rather heavy
Angular Universal front-end that has throughput issues of its own,
while I (and many others) ponder how to fix *those* issues.

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
library.indianapolis.iu.edu


signature.asc
Description: PGP signature

RE: Issue with the log4j2.xml

[Burle, Saicharan]

Hi Team,

Can I get any update on this request? I have responded to the queries asked, 
PFA email for the same.

Regards,
Saicharan Burle
Lead Infrastructure Engineer
Chief Technology Office | Foundational Hosting Platform | Middleware Product 
Engineering
G=ETI DPS Middleware Prod Eng

Block B1A Divyasree Orion | Hyderabad - 500081 | India
MAC O2806-010 | Cell +91 9966964321

MPE Service Desk: 
https://devops-servicedesk.wellsfargo.net/servicedesk/customer/portal/14/group/385?groupId=385
Enterprise Middleware Services: 
https://wim-wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE&title=Middleware+Product+Engineering+Team+Home
Normal Working hours: 4:00 AM to 1:00 PM EST
Upcoming PTO:
Be Green, Leave it on Screen!!!
[http://f.connect.wellsfargoemail.com/i/38/2082566297/EML-224071_4726581_6304406_logo-stagecoach-eml_511x80.png]



From: Christopher Schultz 
Sent: Friday, July 26, 2024 7:24 PM
To: users@tomcat.apache.org
Subject: Re: Issue with the log4j2.xml

Saicharan, On 7/26/24 09: 36, Burle, Saicharan wrote: > We are observing a 
strange behavior for our app running on the Tomcat9. > > Issue: > We have 2 
applications configured on tomcat9 (Java8) named app1 & app2. There are 2 log


Saicharan,



On 7/26/24 09:36, Burle, Saicharan wrote:

> We are observing a strange behavior for our app running on the Tomcat9.

>

> Issue:

> We have 2 applications configured on tomcat9 (Java8) named app1 & app2. There 
> are 2 log files named a.log & b.log to be created under a path /a/b/c (for 
> app1) & /d/e/f (for app2) and this path is defined in log4j2.xml. The log 
> files a.log & b.log is getting created at /a/b/c for app1 but not for app2 
> and instead it is getting written in default Catalina.out file. Nothing 
> created under /d/e/f.



Where is log4j2.xml located?



Where is log4j2-*.jar located?



How does each application initialize log4j2?



> We tried few steps in our lab server and below are our observations.

>

>

>1.  Got this issue when we are Migrating from RHEL-7 to RHEL-8



Which of the following changed during this migration?



- Java version

- Tomcat version

- Application version/configuration/etc



>2.  The same war file for app1 and app2 is working as expected in RHEL-7 
> (a.log and b.log getting created under the path /a/b/c and d/e/f 
> respectively).

>3.  No Issue on RHEL-8, When the war is built locally (created from same 
> branch where Prod war file is) and deployed MANUALLY.

>4.  We got ISSUE on RHEL-8, when the war is built and deployed via UCD.

>5.  No issues in RHEL-7 with UCD build and deploy.

>

> So, we are not sure if it is RHEL-8 or the UCD that is causing this issue. 
> Could you please take a look at it and help with steps to resolve this issue.

 >

> Please let me know in case you need further information.



What is UCD?



If things run under both RHEL-7 and RHEL-8 when building locally, I

would guess that something is wrong with either (a) the data transfer

mechanism or (b) the configuration of the target environment, and the

data-transfer is not relevant.



I would focus on the environmental changes. Look for what files are

where (under the Tomcat deployment, including Tomcat configuration files

and libraries), and the differences between the working-environment and

the non-working environment.



-chris



-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: 
users-h...@tomcat.apache.org


--- Begin Message ---
Hi Chirs, PFB my answers inline: Where is log4j2. xml located? 
/apps/tomcat/deploy///pkg//WEB-INF/classes Where is log4j2-*. 
jar located? /apps/tomcat/deploy//pkg//WEB-INF/lib/

Hi Chirs,

PFB my answers inline:

Where is log4j2.xml located?  
/apps/tomcat/deploy///pkg//WEB-INF/classes





Where is log4j2-*.jar located? 
/apps/tomcat/deploy//pkg//WEB-INF/lib/


How does each application initialize log4j2? Log4j gets initialized during 
server restarts (auto initialized) . From the logs we could see initialization 
is happening for app1 but NOT for app2





Which of the following changed during this migration?



- Java version - No

- Tomcat version - No

- Application version/configuration/etc – No (We have copied entire 
configuration files and folder from RHEL7 to RHEL8 still facing the same issue.)



Only change is from RHEL7 to RHEL8



What is UCD? – Urbancode deploy. A tool that we use for deployment.



If things run under both RHEL-7 and RHEL-8 when building locally, I

would guess that something is wrong with either (a) the data transfer

mechanism or (b) the configuration of the target environment, and the

data-transfer is not relevant.



I would focus on the environmental changes. Look for what files are

where (under the Tomcat deployment, including Tomcat configuration files

and libraries), and the dif

Jakarta EE 11 Release Delayed

[William Crowell]

Good morning,

I received an update from the Jakarta EE Community mailing list this morning 
that the Jakarta EE 11 final release will be pushed out a quarter to deal with 
platform TCK migration to Arquillian/Junit 5. The exact target date is still 
TBD.

I am assuming this also pushes Apache Tomcat 11’s final release as well?

Regards,

William Crowell


This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

Re: Issue with the log4j2.xml

[Sebastian Trost]

Saicharan,

I think you're mistaking this mailing list for a paid support hotline. 
Nobody on this mailing list is getting paid to solve your problem. Chris 
already gave you a couple of hints.


This is what I'm getting from your first e-mail:

RHEL-7 "UCD build and deploy": OK
RHEL-8 "apps built locally and deployed manually": OK
RHEL-8 "UCD build and deploy": NOT OK

To me, your "UCD" (whatever this is) looks to be the culprit here.

Sebastian

On 30.07.2024 16:52, Burle, Saicharan wrote:


Hi Team,

Can I get any update on this request? I have responded to the queries 
asked, PFA email for the same.


*Regards,*

*Saicharan Burle*

*Lead Infrastructure Engineer*

*Chief Technology Office | Foundational Hosting Platform | Middleware 
Product Engineering*


G=ETI DPS Middleware Prod Eng

Block B1A Divyasree Orion | Hyderabad - 500081 | India

MAC O2806-010 | Cell +91 9966964321

MPE Service Desk: 
https://devops-servicedesk.wellsfargo.net/servicedesk/customer/portal/14/group/385?groupId=385 



Enterprise Middleware Services: 
https://wim-wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE&title=Middleware+Product+Engineering+Team+Home 



*Normal Working hours: 4:00 AM to 1:00 PM EST*

*Upcoming PTO:*

Be Green, Leave it on Screen!!!

*From:*Christopher Schultz 
*Sent:* Friday, July 26, 2024 7:24 PM
*To:* users@tomcat.apache.org
*Subject:* Re: Issue with the log4j2.xml

Saicharan, On 7/26/24 09: 36, Burle, Saicharan wrote: > We are 
observing a strange behavior for our app running on the Tomcat9. > > 
Issue: > We have 2 applications configured on tomcat9 (Java8) named 
app1 & app2. There are 2 log


Saicharan,
On 7/26/24 09:36, Burle, Saicharan wrote:
> We are observing a strange behavior for our app running on the Tomcat9.
> 
> Issue:

> We have 2 applications configured on tomcat9 (Java8) named app1 & app2. There are 2 log 
files named a.log & b.log to be created under a path /a/b/c (for app1) & /d/e/f (for app2) 
and this path is defined in log4j2.xml. The log files a.log & b.log is getting created at 
/a/b/c for app1 but not for app2 and instead it is getting written in default Catalina.out file. 
Nothing created under /d/e/f.
Where is log4j2.xml located?
Where is log4j2-*.jar located?
How does each application initialize log4j2?
> We tried few steps in our lab server and below are our observations.
> 
> 
>    1.  Got this issue when we are Migrating from RHEL-7 to RHEL-8

Which of the following changed during this migration?
- Java version
- Tomcat version
- Application version/configuration/etc
>    2.  The same war file for app1 and app2 is working as expected in RHEL-7 
(a.log and b.log getting created under the path /a/b/c and d/e/f respectively).
>    3.  No Issue on RHEL-8, When the war is built locally (created from same 
branch where Prod war file is) and deployed MANUALLY.
>    4.  We got ISSUE on RHEL-8, when the war is built and deployed via UCD.
>    5.  No issues in RHEL-7 with UCD build and deploy.
> 
> So, we are not sure if it is RHEL-8 or the UCD that is causing this issue. Could you please take a look at it and help with steps to resolve this issue.

>
> Please let me know in case you need further information.
What is UCD?
If things run under both RHEL-7 and RHEL-8 when building locally, I
would guess that something is wrong with either (a) the data transfer
mechanism or (b) the configuration of the target environment, and the
data-transfer is not relevant.
I would focus on the environmental changes. Look for what files are
where (under the Tomcat deployment, including Tomcat configuration files
and libraries), and the differences between the working-environment and
the non-working environment.
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: how to use provider with java 18 , différent from java 11

[Christopher Schultz]

Aughra,

On 7/27/24 10:32, aughra wrote:

Hello everyone,

Maybe this question has been asked many times, but I have a problem that 
I can't find a solution for,.


To summarize, I have: A Java program WAR on Tomcat, and Tomcat must 
contain a provider to call an encryption module to obtain keys.


The Tomcat version is 10.1.26

In Java 11, to specify where the pass/crt of the module is located, I 
specify it in the java.security file as follows:


properties

Copier le code

#

# List of providers and their preference orders (see above):

#

security.provider.1=SUN

security.provider.2=SunRsaSign

security.provider.3=SunEC

security.provider.4=SunJSSE

security.provider.5=SunJCE

security.provider.6=SunJGSS

security.provider.7=SunSASL

security.provider.8=XMLDSig

security.provider.9=SunPCSC

security.provider.10=JdkLDAP

security.provider.11=JdkSASL

security.provider.12=SunPKCS11 
/opt/tomcat/webapps/prgmwar/WEB-INF/classes/Crypto.properties


The Crypto.propertiesfile contains the name of the provider to use and 
the library to call the Sun PKCS#11 provider:


library=lib.so

name=cryptto

Which is used in the application parameter (SunPKCS11 is normal, WL, and 
the documentation states it):


module.titi.providerName=SunPKCS11-cryptto

However, it is clearly stated that in Java 17, this solution is no 
longer supported, and it must be done differently; otherwise, I get 
errors in Catalina.


It is stated not to put the path in the java.securityfile and to leave:

security.provider.12=SunPKCS11

I have tried several solutions, putting the path in setenv, in 
catalina.shwith the option:


JAVA_OPTS="$JAVA_OPTS -Djava.security.properties=file:/toto.file"

CATALINA_OPTS="$CATALINA_OPTS -Djava.security.properties=file:/toto.file"

With the file containing the path: toto.file:

security.provider.12=SunPKCS11 
/opt/tomcat/webapps/prgmwar/WEB-INF/classes/Crypto.properties


Almost everywhere I could put it.

But in catalina.out, I still get the same error with Tomcat:

/opt/tomcat/webapps/prgmwar/WEB-INF/classes/appli.properties-> it loads 
the application


HSM-SERVICE|WARN|http-nio-8080-exec-1||GET 
appli||SecurityModuleFactory|Module appli ignored: error during 
initialization


When it searches for the provider, it can't find it because it doesn't 
have: security.provider.12= SunPKCS11 
/opt/tomcat/webapps/prgmwar/WEB-INF/classes/Crypto.propertiesProvider 
"SunPKCS11-crypt" unknown


However, it works very well in a normal Java JAR (without Tomcat), for 
example, to list the keys, because I directly list the provider's 
location via the -Doption of Java:


DEBUG [main] (SecurityModuleFactory.java:112) - Loading crypt module.

DEBUG [main] (SecurityModuleFactory.java:125) - -> SunPKCS11-crypt

Security.java:125)

DEBUG [main] (SecurityModule.java:59) - Creating HardwareSecurityModule 
crypt


crypt - Beginning listKeys

(SecurityModule.java:121) - name = riri

So my library works...

*My question is, how do we do in Java 17 what I did in Java 8 in : 
java.security:*


*security.provider.12=SunPKCS11 
/opt/tomcat/webapps/prgmwar/WEB-INF/classes/Crypto.properties***


Please don't look for any flaws in the variable names as I have just 
changed them.


How are you launching Tomcat?

Setting CATALINA_OPTS in setenv.sh is, I believe, the correct way to do 
this, but it depends upon how Tomcat is launched. Some environments 
ignore the setenv.sh script.


Can you use 'ps' or similar to see the full command-line used to launch 
the JVM and confirm your -D appears there? If it's not there, you have 
made a small mistake somewhere or Tomcat isn't being started using 
catalina.sh.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Which is better: many concurrent requests or a deep backlog?

[Christopher Schultz]

To whom it may concern,

On 7/30/24 10:33, mw...@iu.edu wrote:

For various reasons I have Tomcat behind Apache HTTPD via
mod_proxy_ajp, and I'm looking for good ways to think about the proxy
connection pool.  I'm wondering if I should let HTTPD make as many
connections as it wants, or set a maximum pool size and let requests
wait in the proxy.  This is probably a complex issue and I want to be
sure I'm considering the right aspects.


From the user's perspective, I think it doesn't matter which choice you 
make.



I think I do want persistent proxy connections.


AJP expects to use persistent connections.

This seems to work well and it saves setup/teardown work.  So I have 
"ProxyPass...enablereuse=on" (which I think is default, but I hate 
relying on defaults for things I care about).


Tomcat v9.0.90, HTTPD 2.4.62.  There are no obvious problems at the
moment.  I just want to be sure I'm doing all I can to keep this
back-end service from ever being a bottleneck for a rather heavy
Angular Universal front-end that has throughput issues of its own,
while I (and many others) ponder how to fix *those* issues.


No matter which strategy you choose, I suspect the bottleneck will 
always be the same thing: your application (or some proxy thereof, such 
as a database, back-end API, etc.).


Do you really care if connections are queuing-up in httpd versus Tomcat? 
One could argue that it's slightly less wasteful to have them queue-up 
on the httpd side but not really in terms of actual resource usage.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Reg: tomcat CPU spikes

[Christopher Schultz]

Mark,

On 7/29/24 05:37, Mark Thomas wrote:

On 26/07/2024 22:19, Jalaj Asher wrote:

Thanks Christopher.
But can we also consider allowing caching of different types as 
caching jar files is very valuable as that avoids jar scans real time 
when the production is under load .


But trying to cache static content, which can be cached separately on 
client end, or jsps which are compiled and cached in work folder don’t 
need to be loaded in memory as they force a significant increase in 
memory utilization.


Sounds like you need to provide a custom 
org.apache.catalina.WebResourceRoot$CacheStrategy implementation.


See https://tomcat.apache.org/tomcat-11.0-doc/config/resources.html 
towards the end of the page and 
https://tomcat.apache.org/tomcat-11.0-doc/api/org/apache/catalina/WebResourceRoot.CacheStrategy.html


Oh, that's interesting. So for example you could implement something 
like this:


@Override

public boolean noCache(String path) {
return !path.endsWith(".jar");
}

?

This would only cache JAR files (and case-sensitively I might add).

-chris


-Original Message-
From: Christopher Schultz 
Sent: Monday, July 22, 2024 12:35 PM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source outside 
of eClinicalWorks. Always use caution when opening attachments, 
clicking links, or when responding to this email. If you feel this is 
a phishing scam, please use the Phish Alert Report button in Outlook.



Jalaj,

On 7/19/24 14:28, Jalaj Asher wrote:

This is the warning message we get when cachingAllowed is not set to
false

org.apache.catalina.webresources.Cache.getResource Unable to add the 
resource at [/WEB-INF/classes/] to the cache for web application 
[/x] because there was insufficient free space available after 
evicting expired cache entries - consider increasing the maximum size 
of the cache.


Okay, I see it. Specifically, it is a WARN message which is usually 
not suppressed in a production configuration.


@markt @remm what do you think about making this another of those 
"WARN the first time, DEBUG thereafter" kinds of messages?


It seems like if a cache is full, the operator SHOULD get a 
notification, but if the cache is thrashing, printing an error a huge 
number of times doesn't seem like it's terribly helpful. It just fills 
the disk.


-chris


-Original Message-
From: Jalaj Asher
Sent: Tuesday, July 16, 2024 1:30 PM
To: Tomcat Users List 
Subject: RE: Reg: tomcat CPU spikes


space". Which was very quickly filling up our disk space as well as
increasing disk IO causing latency concerns.
1. Also interesting. Can you post one of those messages here? Was 
there a stack trace shown or just the warning?


  It is just the warning. No stack trace. I will work on 
recreating this since all our environments has this disabled.


2. Interesting. How much static content do you have? This seems like 
a good use-case for a reverse-proxy to handle your static    content 
for you.
We have not collated the complete size of it. But are reasons we 
cannot do that.


Also I was reviewing some older heap dumps and I could see that the 
jars are getting cached in tomcat even with cachingAllowed=false.


Also this is not a consistent issue once it happens it takes sometime 
for the stack to go away as well as post tomcat reboots the problem 
goes away with the same settings and we do see that the wars are 
getting deployed during tomcat startup as well.


Regards

Jalaj P Asher


-Original Message-
From: Christopher Schultz 
Sent: Tuesday, July 16, 2024 10:05 AM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source 
outside of eClinicalWorks. Always use caution when opening 
attachments, clicking links, or when responding to this email. If you 
feel this is a phishing scam, please use the Phish Alert Report 
button in Outlook.



Jalaj,

On 7/15/24 18:18, Jalaj Asher wrote:

We ran into 2 issues
1. We needed to allocate significant amount of -XMX  for heap space,
if we allowed caching, since increasing memory by a few hundred MB as
well was not enough.
Interesting. How much static content do you have? This seems like a 
good use-case for a reverse-proxy to handle your static content for you.



2. Also with the setting being  enabled, it generated logs stating
"could not add a resource  as there wasn’t enough
space". Which was very quickly filling up our disk space as well as
increasing disk IO causing latency concerns.
Also interesting. Can you post one of those messages here? Was there 
a stack trace shown or just the warning?


-chris


-Original Message-
From: Christopher Schultz 
Sent: Monday, July 15, 2024 4:19 PM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source 
outside of eClinicalWorks. Always use caution when opening 
attachm

Re: Jakarta EE 11 Release Delayed

[Mark Thomas]

On 30/07/2024 15:53, William Crowell wrote:

Good morning,

I received an update from the Jakarta EE Community mailing list this morning 
that the Jakarta EE 11 final release will be pushed out a quarter to deal with 
platform TCK migration to Arquillian/Junit 5. The exact target date is still 
TBD.

I am assuming this also pushes Apache Tomcat 11’s final release as well?


I don't see any reason for it to do that.

Tomact 11 milestones are already at beta meaning:

- the specifications Tomcat 11 implements have been released

- Tomcat fully implmements the specifications (and passes the Servlet,
  Pages, EL, WebSocket and Annotations TCKs)

My plan was to see how the August release went and - if things went well 
- start a discussion on the dev@ list about moving to stable releases 
for Tomcat 11.


In case by final release you meant EOL date, that will be driven by the 
release date for Tomcat 14 since we support 3 major releases in 
parallel. It is hard to predict that far in the future but past major 
Tomact releases have typically been support for ~10 years. At this point 
I dont see any reason why Tomcat 11 woudl be different.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org