RE: Issue with the log4j2.xml

2024-07-29 Thread Burle, Saicharan 
Hi Chirs,

PFB my answers inline:

Where is log4j2.xml located?  
/apps/tomcat/deploy///pkg//WEB-INF/classes





Where is log4j2-*.jar located? 
/apps/tomcat/deploy//pkg//WEB-INF/lib/


How does each application initialize log4j2? Log4j gets initialized during 
server restarts (auto initialized) . From the logs we could see initialization 
is happening for app1 but NOT for app2





Which of the following changed during this migration?



- Java version - No

- Tomcat version - No

- Application version/configuration/etc – No (We have copied entire 
configuration files and folder from RHEL7 to RHEL8 still facing the same issue.)



Only change is from RHEL7 to RHEL8



What is UCD? – Urbancode deploy. A tool that we use for deployment.



If things run under both RHEL-7 and RHEL-8 when building locally, I

would guess that something is wrong with either (a) the data transfer

mechanism or (b) the configuration of the target environment, and the

data-transfer is not relevant.



I would focus on the environmental changes. Look for what files are

where (under the Tomcat deployment, including Tomcat configuration files

and libraries), and the differences between the working-environment and

the non-working environment. – We have copied entire working directory from 
RHEL7 to RHEL8



Regards,
Saicharan Burle
Lead Infrastructure Engineer
Chief Technology Office | Foundational Hosting Platform | Middleware Product 
Engineering
G=ETI DPS Middleware Prod Eng

Block B1A Divyasree Orion | Hyderabad - 500081 | India
MAC O2806-010 | Cell +91 9966964321

MPE Service Desk: 
https://devops-servicedesk.wellsfargo.net/servicedesk/customer/portal/14/group/385?groupId=385
Enterprise Middleware Services: 
https://wim-wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE&title=Middleware+Product+Engineering+Team+Home
Normal Working hours: 4:00 AM to 1:00 PM EST
Upcoming PTO:
Be Green, Leave it on Screen!!!
[http://f.connect.wellsfargoemail.com/i/38/2082566297/EML-224071_4726581_6304406_logo-stagecoach-eml_511x80.png]



From: Christopher Schultz 
Sent: Friday, July 26, 2024 7:24 PM
To: users@tomcat.apache.org
Subject: Re: Issue with the log4j2.xml

Saicharan, On 7/26/24 09: 36, Burle, Saicharan wrote: > We are observing a 
strange behavior for our app running on the Tomcat9. > > Issue: > We have 2 
applications configured on tomcat9 (Java8) named app1 & app2. There are 2 log


Saicharan,



On 7/26/24 09:36, Burle, Saicharan wrote:

> We are observing a strange behavior for our app running on the Tomcat9.

>

> Issue:

> We have 2 applications configured on tomcat9 (Java8) named app1 & app2. There 
> are 2 log files named a.log & b.log to be created under a path /a/b/c (for 
> app1) & /d/e/f (for app2) and this path is defined in log4j2.xml. The log 
> files a.log & b.log is getting created at /a/b/c for app1 but not for app2 
> and instead it is getting written in default Catalina.out file. Nothing 
> created under /d/e/f.



Where is log4j2.xml located?



Where is log4j2-*.jar located?



How does each application initialize log4j2?



> We tried few steps in our lab server and below are our observations.

>

>

>1.  Got this issue when we are Migrating from RHEL-7 to RHEL-8



Which of the following changed during this migration?



- Java version

- Tomcat version

- Application version/configuration/etc



>2.  The same war file for app1 and app2 is working as expected in RHEL-7 
> (a.log and b.log getting created under the path /a/b/c and d/e/f 
> respectively).

>3.  No Issue on RHEL-8, When the war is built locally (created from same 
> branch where Prod war file is) and deployed MANUALLY.

>4.  We got ISSUE on RHEL-8, when the war is built and deployed via UCD.

>5.  No issues in RHEL-7 with UCD build and deploy.

>

> So, we are not sure if it is RHEL-8 or the UCD that is causing this issue. 
> Could you please take a look at it and help with steps to resolve this issue.

 >

> Please let me know in case you need further information.



What is UCD?



If things run under both RHEL-7 and RHEL-8 when building locally, I

would guess that something is wrong with either (a) the data transfer

mechanism or (b) the configuration of the target environment, and the

data-transfer is not relevant.



I would focus on the environmental changes. Look for what files are

where (under the Tomcat deployment, including Tomcat configuration files

and libraries), and the differences between the working-environment and

the non-working environment.



-chris



-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: 
users-h...@tomcat.apache.org

Re: Reg: tomcat CPU spikes

2024-07-29 Thread Mark Thomas 

On 26/07/2024 22:19, Jalaj Asher wrote:

Thanks Christopher.
But can we also consider allowing caching of different types as caching jar 
files is very valuable as that avoids jar scans real time when the production 
is under load .

But trying to cache static content, which can be cached separately on client 
end, or jsps which are compiled and cached in work folder don’t need to be 
loaded in memory as they force a significant increase in memory utilization.


Sounds like you need to provide a custom 
org.apache.catalina.WebResourceRoot$CacheStrategy implementation.


See https://tomcat.apache.org/tomcat-11.0-doc/config/resources.html 
towards the end of the page and 
https://tomcat.apache.org/tomcat-11.0-doc/api/org/apache/catalina/WebResourceRoot.CacheStrategy.html


Mark




Regards

Jalaj P Asher

-Original Message-
From: Christopher Schultz 
Sent: Monday, July 22, 2024 12:35 PM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source outside of 
eClinicalWorks. Always use caution when opening attachments, clicking links, or 
when responding to this email. If you feel this is a phishing scam, please use 
the Phish Alert Report button in Outlook.


Jalaj,

On 7/19/24 14:28, Jalaj Asher wrote:

This is the warning message we get when cachingAllowed is not set to
false

org.apache.catalina.webresources.Cache.getResource Unable to add the resource 
at [/WEB-INF/classes/] to the cache for web application [/x] because there 
was insufficient free space available after evicting expired cache entries - 
consider increasing the maximum size of the cache.


Okay, I see it. Specifically, it is a WARN message which is usually not 
suppressed in a production configuration.

@markt @remm what do you think about making this another of those "WARN the first 
time, DEBUG thereafter" kinds of messages?

It seems like if a cache is full, the operator SHOULD get a notification, but 
if the cache is thrashing, printing an error a huge number of times doesn't 
seem like it's terribly helpful. It just fills the disk.

-chris


-Original Message-
From: Jalaj Asher
Sent: Tuesday, July 16, 2024 1:30 PM
To: Tomcat Users List 
Subject: RE: Reg: tomcat CPU spikes


space". Which was very quickly filling up our disk space as well as
increasing disk IO causing latency concerns.

1. Also interesting. Can you post one of those messages here? Was there a stack 
trace shown or just the warning?

  It is just the warning. No stack trace. I will work on recreating this 
since all our environments has this disabled.

2. Interesting. How much static content do you have? This seems like a good 
use-case for a reverse-proxy to handle your staticcontent for you.
We have not collated the complete size of it. But are reasons we cannot do that.

Also I was reviewing some older heap dumps and I could see that the jars are 
getting cached in tomcat even with cachingAllowed=false.

Also this is not a consistent issue once it happens it takes sometime for the 
stack to go away as well as post tomcat reboots the problem goes away with the 
same settings and we do see that the wars are getting deployed during tomcat 
startup as well.

Regards

Jalaj P Asher


-Original Message-
From: Christopher Schultz 
Sent: Tuesday, July 16, 2024 10:05 AM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source outside of 
eClinicalWorks. Always use caution when opening attachments, clicking links, or 
when responding to this email. If you feel this is a phishing scam, please use 
the Phish Alert Report button in Outlook.


Jalaj,

On 7/15/24 18:18, Jalaj Asher wrote:

We ran into 2 issues
1. We needed to allocate significant amount of -XMX  for heap space,
if we allowed caching, since increasing memory by a few hundred MB as
well was not enough.

Interesting. How much static content do you have? This seems like a good 
use-case for a reverse-proxy to handle your static content for you.


2. Also with the setting being  enabled, it generated logs stating
"could not add a resource  as there wasn’t enough
space". Which was very quickly filling up our disk space as well as
increasing disk IO causing latency concerns.

Also interesting. Can you post one of those messages here? Was there a stack 
trace shown or just the warning?

-chris


-Original Message-
From: Christopher Schultz 
Sent: Monday, July 15, 2024 4:19 PM
To: users@tomcat.apache.org
Subject: Re: Reg: tomcat CPU spikes

Attention! - This email has originated from an External Source outside of 
eClinicalWorks. Always use caution when opening attachments, clicking links, or 
when responding to this email. If you feel this is a phishing scam, please use 
the Phish Alert Report button in Outlook.


Jalaj,

On 7/15/24 15:03, Jalaj Asher wrote:

Yeah I was wondering the same as this has been in place since a few
years now atleas

RE: Issue with the log4j2.xml

2024-07-29 Thread Burle, Saicharan 
Hi Team,

Can I get an update on the below issue?

Thanks,
Saicharan Burle

From: Burle, Saicharan 
Sent: Monday, July 29, 2024 2:29 PM
To: Tomcat Users List ; ch...@christopherschultz.net
Subject: RE: Issue with the log4j2.xml

Hi Chirs, PFB my answers inline: Where is log4j2. xml located? 
/apps/tomcat/deploy///pkg//WEB-INF/classes Where is log4j2-*. 
jar located? /apps/tomcat/deploy//pkg//WEB-INF/lib/


Hi Chirs,



PFB my answers inline:



Where is log4j2.xml located?  
/apps/tomcat/deploy///pkg//WEB-INF/classes











Where is log4j2-*.jar located? 
/apps/tomcat/deploy//pkg//WEB-INF/lib/





How does each application initialize log4j2? Log4j gets initialized during 
server restarts (auto initialized) . From the logs we could see initialization 
is happening for app1 but NOT for app2











Which of the following changed during this migration?







- Java version - No



- Tomcat version - No



- Application version/configuration/etc – No (We have copied entire 
configuration files and folder from RHEL7 to RHEL8 still facing the same issue.)







Only change is from RHEL7 to RHEL8







What is UCD? – Urbancode deploy. A tool that we use for deployment.







If things run under both RHEL-7 and RHEL-8 when building locally, I



would guess that something is wrong with either (a) the data transfer



mechanism or (b) the configuration of the target environment, and the



data-transfer is not relevant.







I would focus on the environmental changes. Look for what files are



where (under the Tomcat deployment, including Tomcat configuration files



and libraries), and the differences between the working-environment and



the non-working environment. – We have copied entire working directory from 
RHEL7 to RHEL8







Regards,

Saicharan Burle

Lead Infrastructure Engineer

Chief Technology Office | Foundational Hosting Platform | Middleware Product 
Engineering

G=ETI DPS Middleware Prod Eng



Block B1A Divyasree Orion | Hyderabad - 500081 | India

MAC O2806-010 | Cell +91 9966964321



MPE Service Desk: 
https://urldefense.com/v3/__https://devops-servicedesk.wellsfargo.net/servicedesk/customer/portal/14/group/385?groupId=385__;!!F9svGWnIaVPGSwU!ugowykuW49XDLDyMAeXtYU9fbcaFsrI0wY2jIpVespq1g7rydvv4Omnkl-F_tAEKH1j73NkWMX_JqLo4XljHKAP4JzIrVA1Nsb_jiPZYqikM$

Enterprise Middleware Services: 
https://wim-wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE&title=Middleware+Product+Engineering+Team+Home

Normal Working hours: 4:00 AM to 1:00 PM EST

Upcoming PTO:

Be Green, Leave it on Screen!!!

[https://urldefense.com/v3/__http://f.connect.wellsfargoemail.com/i/38/2082566297/EML-224071_4726581_6304406_logo-stagecoach-eml_511x80.png__;!!F9svGWnIaVPGSwU!ugowykuW49XDLDyMAeXtYU9fbcaFsrI0wY2jIpVespq1g7rydvv4Omnkl-F_tAEKH1j73NkWMX_JqLo4XljHKAP4JzIrVA1Nsb_jiCficwrE$]







From: Christopher Schultz 
mailto:ch...@christopherschultz.net>>

Sent: Friday, July 26, 2024 7:24 PM

To: users@tomcat.apache.org

Subject: Re: Issue with the log4j2.xml



Saicharan, On 7/26/24 09: 36, Burle, Saicharan wrote: > We are observing a 
strange behavior for our app running on the Tomcat9. > > Issue: > We have 2 
applications configured on tomcat9 (Java8) named app1 & app2. There are 2 log





Saicharan,







On 7/26/24 09:36, Burle, Saicharan wrote:



> We are observing a strange behavior for our app running on the Tomcat9.



>



> Issue:



> We have 2 applications configured on tomcat9 (Java8) named app1 & app2. There 
> are 2 log files named a.log & b.log to be created under a path /a/b/c (for 
> app1) & /d/e/f (for app2) and this path is defined in log4j2.xml. The log 
> files a.log & b.log is getting created at /a/b/c for app1 but not for app2 
> and instead it is getting written in default Catalina.out file. Nothing 
> created under /d/e/f.







Where is log4j2.xml located?







Where is log4j2-*.jar located?







How does each application initialize log4j2?







> We tried few steps in our lab server and below are our observations.



>



>



>1.  Got this issue when we are Migrating from RHEL-7 to RHEL-8







Which of the following changed during this migration?







- Java version



- Tomcat version



- Application version/configuration/etc







>2.  The same war file for app1 and app2 is working as expected in RHEL-7 
> (a.log and b.log getting created under the path /a/b/c and d/e/f 
> respectively).



>3.  No Issu