I see the same problem on RHEL 6, but only the thread part of it. It
does not escalate to a heap OOME, so the test "just" fails but does not
hang. I do not have a thread dump there, but on RHEL 6 it also only
happens sng the APR connector, so could well be the same APR Sendfile
thread problem.
Regards,
Rainer
Am 27.01.2019 um 02:18 schrieb Rainer Jung:
When running the unit tests for trunk using Java 11 and APR connector I
run out of memory, first in threads, later in heap. The prpoblematic
tests are TestDefaultServletEncodingWithBom and
TestDefaultServletEncodingWithoutBom.
A thread dumps shows 540 sendfile threads like the following:
[junit] "http-apr-127.0.0.1-auto-540-Sendfile" #8113 daemon prio=5
os_prio=0 cpu=0.34ms elapsed=8031.17s tid=0x00007fd4b8171800 nid=0x1249
in Object.wait() [0x00007fd497313000]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(java.base@11.0.2/Native Method)
[junit] - waiting on <no object reference available>
[junit] at java.lang.Object.wait(java.base@11.0.2/Object.java:328)
[junit] at
org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116)
[junit] - waiting to re-lock in wait() <0x00000000f7b38278> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
[junit]
Is it expected, that those tests use that many threads in parallel?
The later heap memory OOME is probably also due to resource constraints
on that VM. The memory when the OOME happens is:
[junit] Heap
[junit] def new generation total 76800K, used 67531K
[0x00000000f0600000, 0x00000000f5950000, 0x00000000f5950000)
[junit] eden space 68288K, 98% used [0x00000000f0600000,
0x00000000f47f2e38, 0x00000000f48b0000)
[junit] from space 8512K, 0% used [0x00000000f48b0000,
0x00000000f48b0000, 0x00000000f5100000)
[junit] to space 8512K, 0% used [0x00000000f5100000,
0x00000000f5100000, 0x00000000f5950000)
[junit] tenured generation total 170688K, used 170687K
[0x00000000f5950000, 0x0000000100000000, 0x0000000100000000)
[junit] the space 170688K, 99% used [0x00000000f5950000,
0x00000000ffffffd0, 0x0000000100000000, 0x0000000100000000)
[junit] Metaspace used 25248K, capacity 25384K, committed
26624K, reserved 26624K
In addition I see native errors (tcnative 1.2.21):
[junit] Exception in thread "Catalina-utility-1"
java.lang.OutOfMemoryError: unable to create native thread: possibly out
of memory or process/resource limits reached
[junit] at java.base/java.lang.Thread.start0(Native Method)
[junit] [156.313s][warning][os,thread] Failed to start thread -
pthread_create failed (EAGAIN) for attributes: stacksize: 1024k,
guardsize: 0k, detached.
[junit] [156.314s][warning][os,thread] Failed to start thread -
pthread_create failed (EAGAIN) for attributes: stacksize: 1024k,
guardsize: 0k, detached.
[junit] at java.base/java.lang.Thread.start(Thread.java:803)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit] at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
[junit] at java.base/java.lang.Thread.run(Thread.java:834)
an then later after many of those:
[junit] 26-Jan-2019 23:34:12.475 SEVERE [main]
org.apache.catalina.core.ContainerBase.startInternal A child container
failed during start
[junit] java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
[junit] at
java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit] at
java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
[junit] at
org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
[junit] at
org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
[junit] at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
[junit] at
org.apache.catalina.core.StandardService.startInternal(StandardService.java:423)
[junit] at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
[junit] at
org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:928)
[junit] at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
[junit] at
org.apache.catalina.startup.Tomcat.start(Tomcat.java:455)
[junit] at
org.apache.catalina.startup.TomcatBaseTest$TomcatWithFastSessionIDs.start(TomcatBaseTest.java:845)
I do not know, whether the pthread errors happen after to many threads
were created or are the reson for the many threads to get created.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
