Re: Reg: tomcat CPU spikes
On 22/07/2024 17:35, Christopher Schultz wrote: Jalaj, On 7/19/24 14:28, Jalaj Asher wrote: This is the warning message we get when cachingAllowed is not set to false org.apache.catalina.webresources.Cache.getResource Unable to add the resource at [/WEB-INF/classes/] to the cache for web application [/x] because there was insufficient free space available after evicting expired cache entries - consider increasing the maximum size of the cache. Okay, I see it. Specifically, it is a WARN message which is usually not suppressed in a production configuration. @markt @remm what do you think about making this another of those "WARN the first time, DEBUG thereafter" kinds of messages? It seems like if a cache is full, the operator SHOULD get a notification, but if the cache is thrashing, printing an error a huge number of times doesn't seem like it's terribly helpful. It just fills the disk. I'm not sure. If this happens occasionally, then an occasional message seems reasonable. If the cache is thrashing then that is bad and a slew of log messages highlighting that doesn't seem unreasonable. Is there a valid case where you would want the cache to thrash? Mark -chris -Original Message- From: Jalaj Asher Sent: Tuesday, July 16, 2024 1:30 PM To: Tomcat Users List Subject: RE: Reg: tomcat CPU spikes space". Which was very quickly filling up our disk space as well as increasing disk IO causing latency concerns. 1. Also interesting. Can you post one of those messages here? Was there a stack trace shown or just the warning? It is just the warning. No stack trace. I will work on recreating this since all our environments has this disabled. 2. Interesting. How much static content do you have? This seems like a good use-case for a reverse-proxy to handle your static content for you. We have not collated the complete size of it. But are reasons we cannot do that. Also I was reviewing some older heap dumps and I could see that the jars are getting cached in tomcat even with cachingAllowed=false. Also this is not a consistent issue once it happens it takes sometime for the stack to go away as well as post tomcat reboots the problem goes away with the same settings and we do see that the wars are getting deployed during tomcat startup as well. Regards Jalaj P Asher -Original Message- From: Christopher Schultz Sent: Tuesday, July 16, 2024 10:05 AM To: users@tomcat.apache.org Subject: Re: Reg: tomcat CPU spikes Attention! - This email has originated from an External Source outside of eClinicalWorks. Always use caution when opening attachments, clicking links, or when responding to this email. If you feel this is a phishing scam, please use the Phish Alert Report button in Outlook. Jalaj, On 7/15/24 18:18, Jalaj Asher wrote: We ran into 2 issues 1. We needed to allocate significant amount of -XMX for heap space, if we allowed caching, since increasing memory by a few hundred MB as well was not enough. Interesting. How much static content do you have? This seems like a good use-case for a reverse-proxy to handle your static content for you. 2. Also with the setting being enabled, it generated logs stating "could not add a resource as there wasn’t enough space". Which was very quickly filling up our disk space as well as increasing disk IO causing latency concerns. Also interesting. Can you post one of those messages here? Was there a stack trace shown or just the warning? -chris -Original Message- From: Christopher Schultz Sent: Monday, July 15, 2024 4:19 PM To: users@tomcat.apache.org Subject: Re: Reg: tomcat CPU spikes Attention! - This email has originated from an External Source outside of eClinicalWorks. Always use caution when opening attachments, clicking links, or when responding to this email. If you feel this is a phishing scam, please use the Phish Alert Report button in Outlook. Jalaj, On 7/15/24 15:03, Jalaj Asher wrote: Yeah I was wondering the same as this has been in place since a few years now atleast 4 years since cachingAllowed had some changes in tomcat 8 which was resulting in it caching all static content as well as jsps and jars and our though process was if we have static content being cached on the client end and jsps in the work folder each time on access we don’t need the cache. Does the cache actively hurt you? Is there a way to cache just the jars and not every thing else in memory ? I think the short answer is "no there is not a way to do this" but I may be wrong. The long answer might be "maybe, but you will have to play games with and and maybe some other things to get it working. I would save yourself some complexity and simply enable caching. -chris -Original Message- From: Christopher Schultz Sent: Friday, July 12, 2024 4:02 PM To: users@tomcat.apache.org Subject: Re: Reg: tomcat CPU spikes Attention! - This email has originated from an Externa
Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Ladies and Gentlemen: We still have a chronic Tomcat crashing problem at one of our installations. The weirdest thing about this is that while this is certainly *one* of our heaviest-usage installations, it's not *the* heaviest. We already have Tomcat shutting down and restarting itself every night. And we have the Catalina job and its associated JVM job running in a private subsystem, with a 7G private memory pool before it starts to dip into the base memory pool. And we're launching with (according to catalina.out) -Xms4096m Xmx5120m. Our webapp has integration with M$ Office 365, which this installation uses. The usual pattern when it starts to get into trouble may be connected with that integration. Looking at a typical crash in catalina.out, I see several OAuth2 errors that appear to involve an expired token, producing lengthy (over 50 line) Java stacktraces. Other errors seem to involve messages from graph.microsoft.com involving "item not found," that seem to be connected with email attachment downloads from Office365. Then a NullPointerException is thrown, producing a stacktrace of over 60 lines. Then another Microsoft "item not found," like the previous one. Then a handshaking error. Not sure what the handshaking error is *with.* Then Tomcat runs out of memory in the Java heap space, does a dump, and everything hits the proverbial fan. 4775 lines of catalina.out entries before we manually shut it down with extreme prejudice and restart it, 508 of them before the first out-of-memory error, the rest after. And yes, I've packaged up an excerpt to send to our webapp developers, to see if they can make head or tail of what went wrong. Anybody have any suggestions of what to look for? -- JHHL - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Hello James, > -Ursprüngliche Nachricht- > Von: James H. H. Lampert > Gesendet: Dienstag, 23. Juli 2024 19:44 > An: Midrange Systems Technical Discussion l...@lists.midrange.com>; Tomcat Users List > Betreff: Problems with the most problematic of our Tomcat installations on > IBM Midrange (cross-posted to Midrange and Tomcat Lists) > > Ladies and Gentlemen: > > We still have a chronic Tomcat crashing problem at one of our installations. > > The weirdest thing about this is that while this is certainly *one* of our > heaviest-usage installations, it's not *the* heaviest. > > We already have Tomcat shutting down and restarting itself every night. > And we have the Catalina job and its associated JVM job running in a private > subsystem, with a 7G private memory pool before it starts to dip into the base > memory pool. And we're launching with (according to > catalina.out) -Xms4096m Xmx5120m. > > Our webapp has integration with M$ Office 365, which this installation uses. > > The usual pattern when it starts to get into trouble may be connected with > that integration. Looking at a typical crash in catalina.out, I see several > OAuth2 > errors that appear to involve an expired token, producing lengthy (over 50 > line) Java stacktraces. > > Other errors seem to involve messages from graph.microsoft.com involving > "item not found," that seem to be connected with email attachment > downloads from Office365. > > Then a NullPointerException is thrown, producing a stacktrace of over 60 > lines. > > Then another Microsoft "item not found," like the previous one. > > Then a handshaking error. Not sure what the handshaking error is *with.* > > Then Tomcat runs out of memory in the Java heap space, does a dump, and If you have a dump on disk, this would be a good start for further investigation. I usually use MAT https://eclipse.dev/mat/ to analyze the dump and figure out what is occupying the memory. After an OOM, the application is usually not stable anymore because you don’t know beforehand which threads exited. Therefore, I would focus on the memory issue. > everything hits the proverbial fan. 4775 lines of catalina.out entries before > we > manually shut it down with extreme prejudice and restart it, > 508 of them before the first out-of-memory error, the rest after. > > And yes, I've packaged up an excerpt to send to our webapp developers, to > see if they can make head or tail of what went wrong. > > Anybody have any suggestions of what to look for? > > -- > JHHL > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Reg: tomcat CPU spikes
Mark, On 7/23/24 03:36, Mark Thomas wrote: On 22/07/2024 17:35, Christopher Schultz wrote: Jalaj, On 7/19/24 14:28, Jalaj Asher wrote: This is the warning message we get when cachingAllowed is not set to false org.apache.catalina.webresources.Cache.getResource Unable to add the resource at [/WEB-INF/classes/] to the cache for web application [/x] because there was insufficient free space available after evicting expired cache entries - consider increasing the maximum size of the cache. Okay, I see it. Specifically, it is a WARN message which is usually not suppressed in a production configuration. @markt @remm what do you think about making this another of those "WARN the first time, DEBUG thereafter" kinds of messages? It seems like if a cache is full, the operator SHOULD get a notification, but if the cache is thrashing, printing an error a huge number of times doesn't seem like it's terribly helpful. It just fills the disk. I'm not sure. If this happens occasionally, then an occasional message seems reasonable. If the cache is thrashing then that is bad and a slew of log messages highlighting that doesn't seem unreasonable. Is there a valid case where you would want the cache to thrash? Good question. I could imagine an operator setting a cache size and it's okay if there is turnover within the cache, but actual thrashing is not great. If I am running a web site and I have 1M static documents but only 500 classes or whatever, I might only want to cache those classes and other resources (e.g. properties files) but not the huge bulk of static content, especially if the files are large. Thanks, -chris -Original Message- From: Jalaj Asher Sent: Tuesday, July 16, 2024 1:30 PM To: Tomcat Users List Subject: RE: Reg: tomcat CPU spikes space". Which was very quickly filling up our disk space as well as increasing disk IO causing latency concerns. 1. Also interesting. Can you post one of those messages here? Was there a stack trace shown or just the warning? It is just the warning. No stack trace. I will work on recreating this since all our environments has this disabled. 2. Interesting. How much static content do you have? This seems like a good use-case for a reverse-proxy to handle your static content for you. We have not collated the complete size of it. But are reasons we cannot do that. Also I was reviewing some older heap dumps and I could see that the jars are getting cached in tomcat even with cachingAllowed=false. Also this is not a consistent issue once it happens it takes sometime for the stack to go away as well as post tomcat reboots the problem goes away with the same settings and we do see that the wars are getting deployed during tomcat startup as well. Regards Jalaj P Asher -Original Message- From: Christopher Schultz Sent: Tuesday, July 16, 2024 10:05 AM To: users@tomcat.apache.org Subject: Re: Reg: tomcat CPU spikes Attention! - This email has originated from an External Source outside of eClinicalWorks. Always use caution when opening attachments, clicking links, or when responding to this email. If you feel this is a phishing scam, please use the Phish Alert Report button in Outlook. Jalaj, On 7/15/24 18:18, Jalaj Asher wrote: We ran into 2 issues 1. We needed to allocate significant amount of -XMX for heap space, if we allowed caching, since increasing memory by a few hundred MB as well was not enough. Interesting. How much static content do you have? This seems like a good use-case for a reverse-proxy to handle your static content for you. 2. Also with the setting being enabled, it generated logs stating "could not add a resource as there wasn’t enough space". Which was very quickly filling up our disk space as well as increasing disk IO causing latency concerns. Also interesting. Can you post one of those messages here? Was there a stack trace shown or just the warning? -chris -Original Message- From: Christopher Schultz Sent: Monday, July 15, 2024 4:19 PM To: users@tomcat.apache.org Subject: Re: Reg: tomcat CPU spikes Attention! - This email has originated from an External Source outside of eClinicalWorks. Always use caution when opening attachments, clicking links, or when responding to this email. If you feel this is a phishing scam, please use the Phish Alert Report button in Outlook. Jalaj, On 7/15/24 15:03, Jalaj Asher wrote: Yeah I was wondering the same as this has been in place since a few years now atleast 4 years since cachingAllowed had some changes in tomcat 8 which was resulting in it caching all static content as well as jsps and jars and our though process was if we have static content being cached on the client end and jsps in the work folder each time on access we don’t need the cache. Does the cache actively hurt you? Is there a way to cache just the jars and not every thing else in memory ? I think the short answer is "no there
Re: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Thomas, On 7/23/24 13:44, James H. H. Lampert wrote: Ladies and Gentlemen: We still have a chronic Tomcat crashing problem at one of our installations. The weirdest thing about this is that while this is certainly *one* of our heaviest-usage installations, it's not *the* heaviest. We already have Tomcat shutting down and restarting itself every night. And we have the Catalina job and its associated JVM job running in a private subsystem, with a 7G private memory pool before it starts to dip into the base memory pool. And we're launching with (according to catalina.out) -Xms4096m Xmx5120m. Might I make a few recommendations and comments, here? 1. Don't bother setting the min and max heap sizes different from each other. Assuming the process is going to grow to the max heap, you're going to need the max anyway so you may as well allocate it all at once up front. 2. What has to fit into that 7GiB private memory pool? Does it include any OS, or is it just the JVM itself? 3. Note that JVM memory requirements aren't limited to the heap. There's plenty of "native memory" that is necessary to run a JVM as well. For example, I have a production application with heap settings -Xms2048M -Xmx2048M and 'ps' tells me that the virtual size of the process is 9.9GiB and the resident size is 2.7GiB. With a ~5GiB heap, you run the risk of hitting your memory limit. If you are getting "OOME heap" then this is not your issue, but it's something to think about. Our webapp has integration with M$ Office 365, which this installation uses. The usual pattern when it starts to get into trouble may be connected with that integration. Looking at a typical crash in catalina.out, I see several OAuth2 errors that appear to involve an expired token, producing lengthy (over 50 line) Java stacktraces. Other errors seem to involve messages from graph.microsoft.com involving "item not found," that seem to be connected with email attachment downloads from Office365. Then a NullPointerException is thrown, producing a stacktrace of over 60 lines. Long stack traces are not uncommon in web-based applications, especially if an application framework is involved and/or if you have many "forward" operations where a request gets forwarded through several components before a response is finally generated. I wouldn't draw too many conclusions from the stack-trace size(s). Then another Microsoft "item not found," like the previous one. Then a handshaking error. Not sure what the handshaking error is *with.* Then Tomcat runs out of memory in the Java heap space, does a dump, and everything hits the proverbial fan. 4775 lines of catalina.out entries before we manually shut it down with extreme prejudice and restart it, 508 of them before the first out-of-memory error, the rest after. And yes, I've packaged up an excerpt to send to our webapp developers, to see if they can make head or tail of what went wrong. Anybody have any suggestions of what to look for? As Thomas suggested, a heap dump would be helpful but if you have already provided that to your application developers there is no need to send it to us. But... if they aren't sure how to read it or aren't sure what it's telling them... perhaps you should send THEM to *us*. It's helpful to know some more about the application. How many users do you typically serve at a time (concurrent logins, not necessarily concurrent requests)? What kind of information gets stored long-term in the application's heap? This would be things such as session storage (per user, should be freed on logout) or caches (typically global, either never freed or some kind of eviction policy like "last 1000 entries"). Sometimes, the application really is just storing too much in memory. I had an application that ran for years very happily with 64MiB of heap. One day it started throwing OOMEs and when we investigated the cause, it turns out we just had more users than we had in the past and we simply needed to resize of heap. (A much later analysis showed that handing much more heap to the memory manager made the GC run a lot more efficiently and "stole" less time away from the application over time. So we run with a 2GiB heap now, even though we don't need nearly that much operating memory.) -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
On 7/23/24 1:25 PM, Christopher Schultz wrote: Thomas, Uh, "James." Thomas was someone who answered earlier. 2. What has to fit into that 7GiB private memory pool? Does it include any OS, or is it just the JVM itself? On an IBM Midrange box, a private memory pool simply provides jobs in a given subsystem with memory that's off-limits to jobs outside that subsystem. If the subsystem description's memory pool usage is set up correctly, it will grab memory from its private pool whenever possible, before going to the public "base" memory pool on the system. This reduces the number of page faults. And based on the back-end server jobs, there are, at last count, 58 users currently signed on, each one with a dedicated connection to a back-end server job. (The webapp is itself a client to a back-end server that was originally designed for a desktop Java client.) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
