Re:

2023-11-07 Thread Greg Huber
>> The "i.e. everything NOT under WEB-INF/lib or WEB-INF/classes" is 
irrespective of which resource collection it is in. So JARs >> from 
PostResources won't be cached.


OK sorry missed the not.

Although I am mapping in post resources to a maven project which has a 
resources folder that ends up in the target/classes folder.




ie

/home/devuser/git/mavenproject/myplugin/src/main/resources/template/..

WARNING: Unable to add the resource at 
[/WEB-INF/classes/template/js/jstree/jquery.hotkeys.min.js] to the cache 
for web application [/myapp] because there was insufficient free space 
available after evicting expired cache entries - consider increasing the 
maximum size of the cache

Nov 07, 2023 6:22:52 AM org.apache.catalina.webresources.Cache getResource


Checking the logs, it only complains about /WEB-INF/classes/

I will test and calculate the size of all the target/classes folder 
which are mapped under /WEB-INF/classes in post resources.


Cheers Greg

On 06/11/2023 12:19, Mark Thomas wrote:


On 06/11/2023 10:57, Greg Huber wrote:
 >> The maximum useful size will be the total size of static 
resources (i.e. everything NOT under WEB-INF/lib or WEB-INF/classes).


Since I have nothing in either of these, its all mapped in the 
PostResources,  I can just calculate the size of the jars, and add a 
bit for luck.  (ie 85mb +5mb).🙂


The "i.e. everything NOT under WEB-INF/lib or WEB-INF/classes" is 
irrespective of which resource collection it is in. So JARs from 
PostResources won't be cached.


Mark




Thanks

On 06/11/2023 09:43, Mark Thomas wrote:

On 05/11/2023 17:23, Greg Huber wrote:

Thanks Mark and Chris.





I have not noticed any slowness yet.

There are alot of jars (approx 160), but the target/classes folder 
are my
app's classes that I am working on.  These can change (ie not 
static), so

may be better to switch it off.

Is there anyway to calculate the size needed for the cache setting?


The maximum useful size will be the total size of static resources 
(i.e. everything NOT under WEB-INF/lib or WEB-INF/classes).


The right size is going to be a trade-off between the cost of the 
memory for the cache and the benefits the cache brings. Those 
benefits are going to be application (and hardware) dependent.


Mark




Thanks Greg


On Sun, 5 Nov 2023 at 15:31, Christopher Schultz <
ch...@christopherschultz.net> wrote:


Greg and Mark,

On 11/5/23 09:31, Mark Thomas wrote:

On 05/11/2023 10:18, Greg Huber wrote:

OK thanks, the docs mention "static resource cache" but I could not
find info on what it actually is.


It caches the content of static resources in memory and uses that 
rather

than accessing disk.


I am loading maven jars and /target/classes.

eg:





base="/home/devuser/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar" 


className="org.apache.catalina.webresources.FileResourceSet"
webAppMount="/WEB-INF/lib/commons-logging-1.2.jar" />


As its purely for development guess it makes no difference?


I doubt you'll notice if you disable it.


+1

Since you are using JAR files, the caching won't matter once the 
classes

themselves are loaded-into memory. So you may observe some slowness
early in the lifetime of the web application after deployment, but at
long as your code, etc. isn't trying to re-scan JAR files all the 
time,

etc. then you should be fine.

-chris


On 05/11/2023 10:02, Mark Thomas wrote:

On 04/11/2023 11:03, Greg Huber wrote:

Hello,

I am using the  and  to run tomcat for
debugging my app (and it is pretty awesome).  I am getting the 
cache
warning limit, as it is 10mb, what effect would it have if I 
turned

off the cache ie cachingAllowed="false" rather than having to
increase the limit all the time?


This is one of those "it depends" questions. There are lots of
factors that will influence how effective the cache is. You 
could try

and reason what the impact would be but you will likely get a more
accurate answer, faster by just trying it and measuring the 
impact.


Mark

- 


To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



- 


To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org






-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomca

Re: TLD jar scanning at Tomcat Startup

2023-11-07 Thread Mark Thomas




On 06/11/2023 20:53, charles didonato wrote:

Good Evening,
Tomcat 9.082 on Windows 11.
Tomcat runs as a Windows service.

When I start Tomcat and deploy my war file, it hangs at the following in
the Catalina Log:

06-Nov-2023 15:21:59.819 INFO [main]
org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned
for TLDs yet contained no TLDs. Enable debug logging for this logger for a
complete list of JARs that were scanned but no TLDs were found in them.
Skipping unneeded JARs during scanning can improve startup time and JSP
compilation time.

I see nothing in stderr or stdout

I have enabled the logging in logging.properties
# To see debug messages in TldLocationsCache, uncomment the following line:
org.apache.jasper.compiler.TldLocationsCache.level = FINE

I have also edited the default list of jars to scan as below:

# Default list of JAR files that should be scanned that overrides the
default
# jarsToSkip list above. This is typically used to include a specific JAR
that
# has been excluded by a broad file name pattern in the jarsToSkip list.
# The list of JARs to scan may be over-ridden at a Context level for
individual
# scan types by configuring a JarScanner with a nested JarScanFilter.
tomcat.util.scan.StandardJarScanFilter.jarsToScan=\
log4j-taglib*.jar,\
log4j-web*.jar,\
log4javascript*.jar,\
slf4j-taglib*.jar,\
jstl-1.2.jar,\
CKFinder-2.3.1.jar, \
sitemesh-2.4.2.jar, \
ckeditor-java-core-3.5.3.jar, \
spring-security-taglibs-4.0.3.RELEASE.jar

It is my understanding that the jarsToScan overrides the
jarsToSkip.
I believe I have no override in my app context war file.

I am not seeing the intended behavior of the jar scanning and the tomcat
server never deploys the single war file.
Eventually the server encounters an OOM state and crashes.
Am I configuring something incorrectly?


I don't think the assumption that the issue is related to JAR scanning 
is correct. That is the last thing in the log file that worked. You need 
to be looking at whatever happens next that doesn't work.


A couple of things to try.

Take 3 three dumps ~5 seconds apart once the start-up appears to freeze.

Enable heap dumps on OOME and analyse them in your favourite profiler.

MARK

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re:

2023-11-07 Thread Mark Thomas

On 07/11/2023 08:05, Greg Huber wrote:
 >> The "i.e. everything NOT under WEB-INF/lib or WEB-INF/classes" is 
irrespective of which resource collection it is in. So JARs >> from 
PostResources won't be cached.


OK sorry missed the not.

Although I am mapping in post resources to a maven project which has a 
resources folder that ends up in the target/classes folder.


Ah. More precisely then, JARs and classes from WEB-INF/lib / 
WEB-INF/classes won't be cached but non-class resources in those 
locations will be.


Mark




ie

/home/devuser/git/mavenproject/myplugin/src/main/resources/template/..

WARNING: Unable to add the resource at 
[/WEB-INF/classes/template/js/jstree/jquery.hotkeys.min.js] to the cache 
for web application [/myapp] because there was insufficient free space 
available after evicting expired cache entries - consider increasing the 
maximum size of the cache

Nov 07, 2023 6:22:52 AM org.apache.catalina.webresources.Cache getResource


Checking the logs, it only complains about /WEB-INF/classes/

I will test and calculate the size of all the target/classes folder 
which are mapped under /WEB-INF/classes in post resources.


Cheers Greg

On 06/11/2023 12:19, Mark Thomas wrote:


On 06/11/2023 10:57, Greg Huber wrote:
 >> The maximum useful size will be the total size of static 
resources (i.e. everything NOT under WEB-INF/lib or WEB-INF/classes).


Since I have nothing in either of these, its all mapped in the 
PostResources,  I can just calculate the size of the jars, and add a 
bit for luck.  (ie 85mb +5mb).🙂


The "i.e. everything NOT under WEB-INF/lib or WEB-INF/classes" is 
irrespective of which resource collection it is in. So JARs from 
PostResources won't be cached.


Mark




Thanks

On 06/11/2023 09:43, Mark Thomas wrote:

On 05/11/2023 17:23, Greg Huber wrote:

Thanks Mark and Chris.





I have not noticed any slowness yet.

There are alot of jars (approx 160), but the target/classes folder 
are my
app's classes that I am working on.  These can change (ie not 
static), so

may be better to switch it off.

Is there anyway to calculate the size needed for the cache setting?


The maximum useful size will be the total size of static resources 
(i.e. everything NOT under WEB-INF/lib or WEB-INF/classes).


The right size is going to be a trade-off between the cost of the 
memory for the cache and the benefits the cache brings. Those 
benefits are going to be application (and hardware) dependent.


Mark




Thanks Greg


On Sun, 5 Nov 2023 at 15:31, Christopher Schultz <
ch...@christopherschultz.net> wrote:


Greg and Mark,

On 11/5/23 09:31, Mark Thomas wrote:

On 05/11/2023 10:18, Greg Huber wrote:

OK thanks, the docs mention "static resource cache" but I could not
find info on what it actually is.


It caches the content of static resources in memory and uses that 
rather

than accessing disk.


I am loading maven jars and /target/classes.

eg:






base="/home/devuser/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar"

className="org.apache.catalina.webresources.FileResourceSet"
webAppMount="/WEB-INF/lib/commons-logging-1.2.jar" />


As its purely for development guess it makes no difference?


I doubt you'll notice if you disable it.


+1

Since you are using JAR files, the caching won't matter once the 
classes

themselves are loaded-into memory. So you may observe some slowness
early in the lifetime of the web application after deployment, but at
long as your code, etc. isn't trying to re-scan JAR files all the 
time,

etc. then you should be fine.

-chris


On 05/11/2023 10:02, Mark Thomas wrote:

On 04/11/2023 11:03, Greg Huber wrote:

Hello,

I am using the  and  to run tomcat for
debugging my app (and it is pretty awesome).  I am getting the 
cache
warning limit, as it is 10mb, what effect would it have if I 
turned

off the cache ie cachingAllowed="false" rather than having to
increase the limit all the time?


This is one of those "it depends" questions. There are lots of
factors that will influence how effective the cache is. You 
could try

and reason what the impact would be but you will likely get a more
accurate answer, faster by just trying it and measuring the 
impact.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org






-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apa

Re: Are there any known class loader leaks in Tomcat 9?

2023-11-07 Thread Olaf Kock


On 06.11.23 18:55, William Crowell wrote:

Good afternoon,

I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am 
loading some very large Java classes into Metaspace.  I know this is not good 
practice, but I inherited this library.  These classes have business rules and 
are doing some basic primitive and array manipulations, and I am running out of 
native memory.  I don’t think there are any JNI calls in this code base.

Are there anything existing issues with the Tomcat 9 class loader?  I doubt 
there are but wanted to check.


Hi William,

when you say "I am loading some very large Java classes into Metaspace. 
I know this is not good practice" it does not sound like this inherited 
library is using tomcat's standard way of classloading - please clarify.


Also, nothing in that statement points to anything where a leak on 
tomcat's side would make a difference: Are you experiencing problems 
after reloading a webapp, or just in general operation? You might simply 
need to assign more metaspace to the VM for this library to operate 
properly.


So, this boils down to:

* How does this library load the classes into memory?

* Do you redeploy?

* What's your metaspace (and general memory) setting, and what are the 
conditions under which you run out of memory?


* Do you see any log message that hint at classes that won't be garbage 
collected because of stale references? Tomcat issues those warnings, in 
case you (for example) start your own background threads that keep 
holding the memory allocated to them.


Also: I second your doubt about Tomcat's classloader being the cause for 
your problems - unless proven otherwise.


Olaf




Re:

2023-11-07 Thread Greg Huber
>>  More precisely then, JARs and classes from WEB-INF/lib / 
WEB-INF/classes won't be cached but non-class resources in those 
locations will be.


This app, the cache starts to complain at approx 18mb.

If I calculate the size roughly (right click properties) of all the 
target/class folders = 21.97mb, and the target/classes/com folders = 
1.0648mb.


So the calculated size would be 20.9129 ie 21mb.

...I guess using post resources this way was not in the initial scope, 
but it works great (way better than the previous way).


Thanks Greg

On 07/11/2023 09:12, Mark Thomas wrote:

On 07/11/2023 08:05, Greg Huber wrote:
 >> The "i.e. everything NOT under WEB-INF/lib or WEB-INF/classes" is 
irrespective of which resource collection it is in. So JARs >> from 
PostResources won't be cached.


OK sorry missed the not.

Although I am mapping in post resources to a maven project which has 
a resources folder that ends up in the target/classes folder.


Ah. More precisely then, JARs and classes from WEB-INF/lib / 
WEB-INF/classes won't be cached but non-class resources in those 
locations will be.


Mark




ie

/home/devuser/git/mavenproject/myplugin/src/main/resources/template/..

WARNING: Unable to add the resource at 
[/WEB-INF/classes/template/js/jstree/jquery.hotkeys.min.js] to the 
cache for web application [/myapp] because there was insufficient 
free space available after evicting expired cache entries - consider 
increasing the maximum size of the cache
Nov 07, 2023 6:22:52 AM org.apache.catalina.webresources.Cache 
getResource



Checking the logs, it only complains about /WEB-INF/classes/

I will test and calculate the size of all the target/classes folder 
which are mapped under /WEB-INF/classes in post resources.


Cheers Greg

On 06/11/2023 12:19, Mark Thomas wrote:


On 06/11/2023 10:57, Greg Huber wrote:
 >> The maximum useful size will be the total size of static 
resources (i.e. everything NOT under WEB-INF/lib or WEB-INF/classes).


Since I have nothing in either of these, its all mapped in the 
PostResources,  I can just calculate the size of the jars, and add 
a bit for luck.  (ie 85mb +5mb).🙂


The "i.e. everything NOT under WEB-INF/lib or WEB-INF/classes" is 
irrespective of which resource collection it is in. So JARs from 
PostResources won't be cached.


Mark




Thanks

On 06/11/2023 09:43, Mark Thomas wrote:

On 05/11/2023 17:23, Greg Huber wrote:

Thanks Mark and Chris.





I have not noticed any slowness yet.

There are alot of jars (approx 160), but the target/classes 
folder are my
app's classes that I am working on.  These can change (ie not 
static), so

may be better to switch it off.

Is there anyway to calculate the size needed for the cache setting?


The maximum useful size will be the total size of static resources 
(i.e. everything NOT under WEB-INF/lib or WEB-INF/classes).


The right size is going to be a trade-off between the cost of the 
memory for the cache and the benefits the cache brings. Those 
benefits are going to be application (and hardware) dependent.


Mark




Thanks Greg


On Sun, 5 Nov 2023 at 15:31, Christopher Schultz <
ch...@christopherschultz.net> wrote:


Greg and Mark,

On 11/5/23 09:31, Mark Thomas wrote:

On 05/11/2023 10:18, Greg Huber wrote:
OK thanks, the docs mention "static resource cache" but I 
could not

find info on what it actually is.


It caches the content of static resources in memory and uses 
that rather

than accessing disk.


I am loading maven jars and /target/classes.

eg:





base="/home/devuser/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar" 


className="org.apache.catalina.webresources.FileResourceSet"
webAppMount="/WEB-INF/lib/commons-logging-1.2.jar" />


As its purely for development guess it makes no difference?


I doubt you'll notice if you disable it.


+1

Since you are using JAR files, the caching won't matter once the 
classes

themselves are loaded-into memory. So you may observe some slowness
early in the lifetime of the web application after deployment, 
but at
long as your code, etc. isn't trying to re-scan JAR files all 
the time,

etc. then you should be fine.

-chris


On 05/11/2023 10:02, Mark Thomas wrote:

On 04/11/2023 11:03, Greg Huber wrote:

Hello,

I am using the  and  to run tomcat 
for
debugging my app (and it is pretty awesome). I am getting 
the cache
warning limit, as it is 10mb, what effect would it have if I 
turned

off the cache ie cachingAllowed="false" rather than having to
increase the limit all the time?


This is one of those "it depends" questions. There are lots of
factors that will influence how effective the cache is. You 
could try
and reason what the impact would be but you will likely get a 
more
accurate answer, faster by just trying it and measuring the 
impact.


Mark

- 


To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tom

Re: Are there any known class loader leaks in Tomcat 9?

2023-11-07 Thread William Crowell
Olaf and Sevendu,

Thank you for your replies.  Correct, I sincerely doubt this is a Tomcat class 
loading bug.

I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into 
memory, and it is a single class loader.

I am going to periodically run: jcmd  GC.class_stats

I am only having the issue in general operation and not on a redeploy, and I 
have to restart the server daily.  I will find out more details as to how these 
classes are loaded into memory.

Here is what is going on…

I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on 
it.  It is running JDK 1.8.0_371-b25.  I have min and max JVM heap setting at 
8GB.  JVM arguments are:

-XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC 
-XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 
-XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 
-XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 
-XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail 
-XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime 
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops 
-XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation

The MaxMetaspaceSize is not set, so this means it is unlimited.

After the server comes up and after a short period of time I get a native out 
of memory condition:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing 
reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may 
be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x14fff94b7700
#
# JRE version:  (8.0_371) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 
compressed oops)
# Core dump written. Default location: /hosting/core or core.191803
#

Regards,

William Crowell


From: Olaf Kock 
Date: Tuesday, November 7, 2023 at 4:25 AM
To: users@tomcat.apache.org 
Subject: Re: Are there any known class loader leaks in Tomcat 9?

On 06.11.23 18:55, William Crowell wrote:
> Good afternoon,
>
> I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am 
> loading some very large Java classes into Metaspace.  I know this is not good 
> practice, but I inherited this library.  These classes have business rules 
> and are doing some basic primitive and array manipulations, and I am running 
> out of native memory.  I don’t think there are any JNI calls in this code 
> base.
>
> Are there anything existing issues with the Tomcat 9 class loader?  I doubt 
> there are but wanted to check.

Hi William,

when you say "I am loading some very large Java classes into Metaspace.
I know this is not good practice" it does not sound like this inherited
library is using tomcat's standard way of classloading - please clarify.

Also, nothing in that statement points to anything where a leak on
tomcat's side would make a difference: Are you experiencing problems
after reloading a webapp, or just in general operation? You might simply
need to assign more metaspace to the VM for this library to operate
properly.

So, this boils down to:

* How does this library load the classes into memory?

* Do you redeploy?

* What's your metaspace (and general memory) setting, and what are the
conditions under which you run out of memory?

* Do you see any log message that hint at classes that won't be garbage
collected because of stale references? Tomcat issues those warnings, in
case you (for example) start your own background threads that keep
holding the memory allocated to them.

Also: I second your doubt about Tomcat's classloader being the cause for
your problems - unless proven otherwise.

Olaf




CAUTION: This email originated from outside of the organization. Do not click 
on links or open attachments unless you recognize the sender and know the 
content is safe.


This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.



Re: Are there any known class loader leaks in Tomcat 9?

2023-11-07 Thread Suvendu Sekhar Mondal
Hello William,

On Tue, Nov 7, 2023 at 4:29 PM William Crowell
 wrote:
>
> Olaf and Sevendu,
>
> Thank you for your replies.  Correct, I sincerely doubt this is a Tomcat 
> class loading bug.
>
> I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into 
> memory, and it is a single class loader.
>
> I am going to periodically run: jcmd  GC.class_stats
>
> I am only having the issue in general operation and not on a redeploy, and I 
> have to restart the server daily.  I will find out more details as to how 
> these classes are loaded into memory.
>
> Here is what is going on…
>
> I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on 
> it.  It is running JDK 1.8.0_371-b25.  I have min and max JVM heap setting at 
> 8GB.  JVM arguments are:
>
> -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC 
> -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 
> -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 
> -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 
> -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 
> -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC 
> -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime 
> -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps 
> -XX:+UseCompressedClassPointers -XX:+UseCompressedOops 
> -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
>
> The MaxMetaspaceSize is not set, so this means it is unlimited.
>
> After the server comes up and after a short period of time I get a native out 
> of memory condition:
>
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 8589934592 bytes for 
> committing reserved memory.
> # Possible reasons:
> #   The system is out of physical RAM or swap space
> #   The process is running with CompressedOops enabled, and the Java Heap may 
> be blocking the growth of the native heap
> # Possible solutions:
> #   Reduce memory load on the system
> #   Increase physical memory or swap space
> #   Check if swap backing store is full
> #   Decrease Java heap size (-Xmx/-Xms)
> #   Decrease number of Java threads
> #   Decrease Java thread stack sizes (-Xss)
> #   Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> #  Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x14fff94b7700
> #
> # JRE version:  (8.0_371) (build )
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode 
> linux-amd64 compressed oops)
> # Core dump written. Default location: /hosting/core or core.191803
> #
>

So, it is not a Metaspace OOM. It can happen that Metaspace or some
other segment of JVM is increasing with time. Since you have NMT
enabled(-XX:NativeMemoryTracking=detail), please collect native memory
stats periodically until JVM crashes. After that, if you compare them,
you should see one of the segments is growing with time. Then you can
focus on that area. jcmd  VM.native_memory can help with this.


> Regards,
>
> William Crowell
>
>
> From: Olaf Kock 
> Date: Tuesday, November 7, 2023 at 4:25 AM
> To: users@tomcat.apache.org 
> Subject: Re: Are there any known class loader leaks in Tomcat 9?
>
> On 06.11.23 18:55, William Crowell wrote:
> > Good afternoon,
> >
> > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am 
> > loading some very large Java classes into Metaspace.  I know this is not 
> > good practice, but I inherited this library.  These classes have business 
> > rules and are doing some basic primitive and array manipulations, and I am 
> > running out of native memory.  I don’t think there are any JNI calls in 
> > this code base.
> >
> > Are there anything existing issues with the Tomcat 9 class loader?  I doubt 
> > there are but wanted to check.
>
> Hi William,
>
> when you say "I am loading some very large Java classes into Metaspace.
> I know this is not good practice" it does not sound like this inherited
> library is using tomcat's standard way of classloading - please clarify.
>
> Also, nothing in that statement points to anything where a leak on
> tomcat's side would make a difference: Are you experiencing problems
> after reloading a webapp, or just in general operation? You might simply
> need to assign more metaspace to the VM for this library to operate
> properly.
>
> So, this boils down to:
>
> * How does this library load the classes into memory?
>
> * Do you redeploy?
>
> * What's your metaspace (and general memory) setting, and what are the
> conditions under which you run out of memory?
>
> * Do you see any log message that hint at classes that won't be garbage
> collected because of stale references? Tomcat issues those warnings, in
> case you (for example) start your own background threads that keep
> holding the memory allocated to them.
>
> Also: I second your doubt about Tomcat's class

Chunk size error after upgrading JRE

2023-11-07 Thread Tuukka Ilomäki
We have a very old application running on Tomcat 8.5.90. After upgrading from 
JRE 8.0.252.09 from AdoptOpenJDK to 8u302b08 from Temurin (both pretty old, I 
know, also newer JREs exhibit the same issue) we started having 
NS_ERROR_NET_PARTIAL_TRANSFER in FireFox and ERR_INCOMPLETE_CHUNKED_ENCODING in 
Chrome. I suspect this has something to do with UTF-8 conversion but I have 
troubles figuring out what and why. The application has a filter that sets 
response character encoding as UTF-8. Truncated content looks like is has UTF-8 
content but truncated to length calculated for ANSI. Naturally, I could be 
completely wrong in the assumption.

As JREs are old, it is hard to fathom what could have changed between those 
versions. Apparently there is some configuration or default that has changed 
but I have difficulties in trying to figure out where to start searching.

Any advice greatly appreciated.

Best,
Tuukka



Re: Chunk size error after upgrading JRE

2023-11-07 Thread Mark Thomas

On 07/11/2023 14:05, Tuukka Ilomäki wrote:

We have a very old application running on Tomcat 8.5.90. After upgrading from 
JRE 8.0.252.09 from AdoptOpenJDK to 8u302b08 from Temurin (both pretty old, I 
know, also newer JREs exhibit the same issue) we started having 
NS_ERROR_NET_PARTIAL_TRANSFER in FireFox and ERR_INCOMPLETE_CHUNKED_ENCODING in 
Chrome. I suspect this has something to do with UTF-8 conversion but I have 
troubles figuring out what and why. The application has a filter that sets 
response character encoding as UTF-8. Truncated content looks like is has UTF-8 
content but truncated to length calculated for ANSI. Naturally, I could be 
completely wrong in the assumption.

As JREs are old, it is hard to fathom what could have changed between those 
versions. Apparently there is some configuration or default that has changed 
but I have difficulties in trying to figure out where to start searching.

Any advice greatly appreciated.


Can you recreate the issue with a simple Servlet? Trying to do so may 
shed some light on what is going on and if it doesn't you'll have a 
reproducible test case for a bug report.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org