Re: Tomcat/Java starts using too much memory and not by the heap or non-heap memory

[Brian Braun]

Hello,

It has been a long time since I received the last suggestions to my issue
here on this support list. Since then I decided to stop asking for help and
to "do my homework". To read, to watch YouTube presentations, to spend time
on StackOverflow, etc. So I have spent a lot of time on this and I think I
have learned a lot which is nice.
This is what I have learned lately:

I definitely don't have a leak in my code (or in the libraries I am using,
as far as I understand). And my code is not creating a significant amount
of objects that would use too much memory.
The heap memory (the 3 G1s) and non-heap memory (3 CodeHeaps + compressed
class space + metaspace) together use just using a few hundred MBs and
their usage is steady and normal.
I discovered the JCMD command to perform the native memory tracking. When
running it, after 3-4 days since I started Tomcat, I found out that the
compiler was using hundreds of MB and that is exactly why the Tomcat
process starts abusing the memory! This is what I saw when executing "sudo
jcmd  VM.native_memory scale=MB":

Compiler (reserved=3D340MB, commited=3D340MB)
(arena=3D340MB #10)

All the other categories (Class, Thread, Code, GC, Internal, Symbol, etc)
look normal since they use a low amount of memory and they don't grow.

Then I discovered the Jemalloc tool (http://jemalloc.net) and its jeprof
tool, so I started launching Tomcat using it. Then, after 3-4 days after
Tomcat starts I was able to create some GIF images from the dumps that
Jemalloc creates. The GIF files show the problem: 75-90% of the memory is
being used by some weird activity in the compiler! It seems that something
called "The C2 compile/JIT compiler" starts doing something after 3-4 days,
and that creates the leak. Why after 3-4 days and not sooner? I don't know.
I am attaching the GIF in this email.

Does anybody know how to deal with this? I have been struggling with this
issue already for 3 months. At least now I know that this is a native
memory leak, but at this point I feel lost.

By the way, I'm running my website using Tomcat 9.0.58, Java
"11.0.21+9-post-Ubuntu-0ubuntu122.04", Ubuntu 22.04.03. And I am developing
using Eclipse and compiling my WAR file with a "Compiler compliance
level:11".

Thanks in advance!

Brian



On Mon, Jan 8, 2024 at 10:05 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Brian,
>
> On 1/5/24 17:21, Brian Braun wrote:
> > Hello Chirstopher,
> >
> > First of all: thanks a lot for your responses!
> >
> > On Wed, Jan 3, 2024 at 9:25 AM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> >> Brian,
> >>
> >> On 12/30/23 15:42, Brian Braun wrote:
> >>> At the beginning, this was the problem: The OOM-killer (something that
> I
> >>> never knew existed) killing Tomcat unexpectedly and without any
> >>> explanation
> >>
> >> The explanation is always the same: some application requests memory
> >> from the kernel, which always grants the request(!). When the
> >> application tries to use that memory, the kernel scrambles to physically
> >> allocate the memory on-demand and, if all the memory is gone, it will
> >> pick a process and kill it.
>  >
> > Yes, that was happening to me until I set up the SWAP file and now at
> least
> > the Tomcat process is not being killed anymore.
>
> Swap can get you out of a bind like this, but it will ruin your
> performance. If you care more about stability (and believe me, it's a
> reasonable decision), then leave the swap on. But swap will kill (a)
> performance (b) SSD lifetime and (c) storage/transaction costs depending
> upon your environment. Besides, you either need the memory or you do
> not. It's rare to "sometimes" need the memory.
>
> >> Using a swap file is probably going to kill your performance. What
> >> happens if you make your heap smaller?
>  >
> > Yes, in fact the performance is suffering and that is why I don't
> consider
> > the swap file as a solution.
>
> :D
>
> > I have assigned to -Xmx both small amounts (as small as 300MB) and high
> > amounts (as high as 1GB) and the problem is still present (the Tomcat
> > process grows in memory usage up to 1.5GB combining real memory and swap
> > memory).
>
> Okay, that definitely indicates a problem that needs to be solved.
>
> I've seen things like native ZIP handling code leaking native memory,
> but I know that Tomcat does not leak like that. If you do anything in
> your application that might leave file handles open, it could be
> contributing to the problem.
>
> > As I have explained in another email recently, I think that neither heap
> > usage nor non-heap usage are the problem. I have been monitoring them and
> > their requirements have always stayed low enough, so I could leave the
> -Xms
> > parameter with about 300-400 MB and that would be enough.
>
> Well, between heap and non-heap, that's all the memory. There is no
> non-heap-non-non-heap memory to be counted. Technically stack space is
> the same as "native memory" but usuall

Re: Tomcat/Java starts using too much memory and not by the heap or non-heap memory

[Brian Braun]

Hello,

It has been a long time since I received the last suggestions to my issue
here on this support list. Since then I decided to stop asking for help and
to "do my homework". To read, to watch YouTube presentations, to spend time
on StackOverflow, etc. So I have spent a lot of time on this and I think I
have learned a lot which is nice.
This is what I have learned lately:

I definitely don't have a leak in my code (or in the libraries I am using,
as far as I understand). And my code is not creating a significant amount
of objects that would use too much memory.
The heap memory (the 3 G1s) and non-heap memory (3 CodeHeaps + compressed
class space + metaspace) together use just using a few hundred MBs and
their usage is steady and normal.
I discovered the JCMD command to perform the native memory tracking. When
running it, after 3-4 days since I started Tomcat, I found out that the
compiler was using hundreds of MB and that is exactly why the Tomcat
process starts abusing the memory! This is what I saw when executing "sudo
jcmd  VM.native_memory scale=MB":

Compiler (reserved=3D340MB, commited=3D340MB)
(arena=3D340MB #10)

All the other categories (Class, Thread, Code, GC, Internal, Symbol, etc)
look normal since they use a low amount of memory and they don't grow.

Then I discovered the Jemalloc tool (http://jemalloc.net) and its jeprof
tool, so I started launching Tomcat using it. Then, after 3-4 days after
Tomcat starts I was able to create some GIF images from the dumps that
Jemalloc creates. The GIF files show the problem: 75-90% of the memory is
being used by some weird activity in the compiler! It seems that something
called "The C2 compile/JIT compiler" starts doing something after 3-4 days,
and that creates the leak. Why after 3-4 days and not sooner? I don't know.
I am attaching the GIF in this email.

Does anybody know how to deal with this? I have been struggling with this
issue already for 3 months. At least now I know that this is a native
memory leak, but at this point I feel lost.

By the way, I'm running my website using Tomcat 9.0.58, Java
"11.0.21+9-post-Ubuntu-0ubuntu122.04", Ubuntu 22.04.03. And I am developing
using Eclipse and compiling my WAR file with a "Compiler compliance
level:11".

Thanks in advance!

Brian

On Mon, Jan 8, 2024 at 10:05 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Brian,
>
> On 1/5/24 17:21, Brian Braun wrote:
> > Hello Chirstopher,
> >
> > First of all: thanks a lot for your responses!
> >
> > On Wed, Jan 3, 2024 at 9:25 AM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> >> Brian,
> >>
> >> On 12/30/23 15:42, Brian Braun wrote:
> >>> At the beginning, this was the problem: The OOM-killer (something that
> I
> >>> never knew existed) killing Tomcat unexpectedly and without any
> >>> explanation
> >>
> >> The explanation is always the same: some application requests memory
> >> from the kernel, which always grants the request(!). When the
> >> application tries to use that memory, the kernel scrambles to physically
> >> allocate the memory on-demand and, if all the memory is gone, it will
> >> pick a process and kill it.
>  >
> > Yes, that was happening to me until I set up the SWAP file and now at
> least
> > the Tomcat process is not being killed anymore.
>
> Swap can get you out of a bind like this, but it will ruin your
> performance. If you care more about stability (and believe me, it's a
> reasonable decision), then leave the swap on. But swap will kill (a)
> performance (b) SSD lifetime and (c) storage/transaction costs depending
> upon your environment. Besides, you either need the memory or you do
> not. It's rare to "sometimes" need the memory.
>
> >> Using a swap file is probably going to kill your performance. What
> >> happens if you make your heap smaller?
>  >
> > Yes, in fact the performance is suffering and that is why I don't
> consider
> > the swap file as a solution.
>
> :D
>
> > I have assigned to -Xmx both small amounts (as small as 300MB) and high
> > amounts (as high as 1GB) and the problem is still present (the Tomcat
> > process grows in memory usage up to 1.5GB combining real memory and swap
> > memory).
>
> Okay, that definitely indicates a problem that needs to be solved.
>
> I've seen things like native ZIP handling code leaking native memory,
> but I know that Tomcat does not leak like that. If you do anything in
> your application that might leave file handles open, it could be
> contributing to the problem.
>
> > As I have explained in another email recently, I think that neither heap
> > usage nor non-heap usage are the problem. I have been monitoring them and
> > their requirements have always stayed low enough, so I could leave the
> -Xms
> > parameter with about 300-400 MB and that would be enough.
>
> Well, between heap and non-heap, that's all the memory. There is no
> non-heap-non-non-heap memory to be counted. Technically stack space is
> the same as "native memory" but usually 

Re: Tomcat/Java starts using too much memory and not by the heap or non-heap memory

[Chuck Caldarale]

> On Feb 15, 2024, at 09:04, Brian Braun  wrote:
> 
> I discovered the JCMD command to perform the native memory tracking. When
> running it, after 3-4 days since I started Tomcat, I found out that the
> compiler was using hundreds of MB and that is exactly why the Tomcat
> process starts abusing the memory! This is what I saw when executing "sudo 
> jcmd  VM.native_memory scale=MB":
> 
> Compiler (reserved=3D340MB, commited=3D340MB)
> (arena=3D340MB #10)
> 
> Then I discovered the Jemalloc tool (http://jemalloc.net 
> ) and its jeprof
> tool, so I started launching Tomcat using it. Then, after 3-4 days after
> Tomcat starts I was able to create some GIF images from the dumps that
> Jemalloc creates. The GIF files show the problem: 75-90% of the memory is
> being used by some weird activity in the compiler! It seems that something
> called "The C2 compile/JIT compiler" starts doing something after 3-4 days,
> and that creates the leak. Why after 3-4 days and not sooner? I don't know.


There have been numerous bugs filed with OpenJDK for C2 memory leaks over the 
past few years, mostly related to recompiling certain methods. The C2 compiler 
kicks in when fully optimizing methods, and it may recompile methods after 
internal instrumentation shows that additional performance can be obtained by 
doing so.


> I am attaching the GIF in this email.


Attachments are stripped on this mailing list.


> Does anybody know how to deal with this?


You could disable the C2 compiler temporarily, and just let C1 handle your 
code. Performance will be somewhat degraded, but may well still be acceptable. 
Add the following to the JVM options when you launch Tomcat:

-XX:TieredStopAtLevel=1


> By the way, I'm running my website using Tomcat 9.0.58, Java
> "11.0.21+9-post-Ubuntu-0ubuntu122.04", Ubuntu 22.04.03. And I am developing
> using Eclipse and compiling my WAR file with a "Compiler compliance
> level:11".


You could try a more recent JVM version; JDK 11 was first released over 5 years 
ago, although it is still being maintained.


  - Chuck

ConnectionPool JDBC vs. DBCP

[ESI]

Moin,
in the docpage https://tomcat.apache.org/tomcat-10.1-doc/jdbc-pool.html it says

>The JDBC Connection Pool org.apache.tomcat.jdbc.pool is a replacement or an 
>alternative to the Apache Commons DBCP connection pool.
>
>So why do we need a new connection pool?
>
>Here are a few of the reasons:
>
>Commons DBCP 1.x is single threaded. In order to be thread safe Commons 
> locks the entire pool for short periods during both object allocation and 
> object return. Note that this does not apply to Commons DBCP 2.x.
[...snip...]

I thought that Tomcat uses DBCP 2.x per default, is this document still 
relevant or is it outdated?

Thanks,
  Andreas



This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org