> On Dec 29, 2023, at 19:48, Brian Braun <java08275...@gmail.com> wrote: > > First of all: > Christopher Schultz: You answered an email from me 6 weeks ago. You helped > me a lot with your suggestions. I have done a lot of research and have > learnt a lot since then, so I have been able to rule out a lot of potential > roots for my issue. Because of that I am able to post a new more specific > email. Thanks a lot!!! > > Now, this is my stack: > > - Ubuntu 22.04.3 on x86/64 with 2GM of physical RAM that has been enough > for years.
I presume the “2GM” above should be “2GB”. > - Java 11.0.20.1+1-post-Ubuntu-0ubuntu122.04 / openjdk 11.0.20.1 2023-08-24 > - Tomcat 9.0.58 (JAVA_OPTS="-Djava.awt.headless=true -Xmx1000m -Xms1000m > ......") > - My app, which I developed myself, and has been running without any > problems for years > > Well, a couple of months ago my website/Tomcat/Java started eating more and > more memory about after about 4-7 days. The previous days it uses just a > few hundred MB and is very steady, but then after a few days the memory > usage suddenly grows up to 1.5GB (and then stops growing at that point, > which is interesting). Between these anomalies the RAM usage is fine and > very steady (as it has been for years) and it uses just about 40-50% of the > "Max memory" (according to what the Tomcat Manager server status shows). > The 3 components of G1GC heap memory are steady and low, before and after > the usage grows to 1.5GB, so it is definitely not that the heap starts > requiring more and more memory. I have been using several tools to monitor > that (New Relic, VisualVM and JDK Mission Control) so I'm sure that the > memory usage by the heap is not the problem. > The Non-heaps memory usage is not the problem either. Everything there is > normal, the usage is humble and even more steady. What does the /proc/<tomcat_pid>/maps file show, both before and after the problem occurs? This should give you some idea of what .so library is grabbing the extra memory. (I only have Tomcat installed on macOS at the moment, so I can’t show you an example; I should be able to bring up Tomcat on a Linux box tomorrow.) The output may be long, depending on how fragmented the virtual memory allocations are. > And there are no leaks, I'm sure of that. I have inspected the JVM using > several tools. > > There are no peaks in the number of threads either. The peak is the same > when the memory usage is low and when it requires 1.5GB. It stays the same > all the time. > > I have also reviewed all the scheduled tasks in my app and lowered the > amount of objects they create, which was nice and entertaining. But that is > not the problem, I have analyzed the object creation by all the threads > (and there are many) and the threads created by my scheduled tasks are very > humble in their memory usage, compared to many other threads. > > And I haven't made any relevant changes to my app in the 6-12 months before > this problem started occurring. It is weird that I started having this > problem. Could it be that I received an update in the java version or the > Tomcat version that is causing this problem? > > If neither the heap memory or the Non-heaps memory is the source of the > growth of the memory usage, what could it be? Clearly something is > happening inside the JVM that raises the memory usage. And everytime it > grows, it doesn't decrease. It is like if something suddenly starts > "pushing" the memory usage more and more, until it stops at 1.5GB. > > I think that maybe the source of the problem is the garbage collector. I > haven't used any of the switches that we can use to optimize that, > basically because I don't know what I should do there (if I should at all). > I have also activated the GC log, but I don't know how to analyze it. I doubt that GC is the problem; if it were, it should show up in the GC data, which you say is essentially the same before and after the problem manifests itself.. > I have also increased and decreased the value of "-Xms" parameter and it is > useless. Unrelated to your problem, but for server processes, -Xms should be set to the same value as -Xmx; no sense in thrashing between the two. > Finally, maybe I should add that I activated 4GB of SWAP memory in my > Ubuntu instance so at least my JVM would not be killed my the OS anymore > (since the real memory is just 1.8GB). That worked and now the memory usage > can grow up to 1.5GB without crashing, by using the much slower SWAP > memory, but I still think that this is an abnormal situation. At least you have a workaround, as undesirable as it may be. - Chuck
