Am 28.01.2019 um 11:49 schrieb Rémy Maucherat:
On Mon, Jan 28, 2019 at 11:16 AM Rainer Jung <rainer.j...@kippdata.de>
wrote:
Am 28.01.2019 um 10:50 schrieb Rémy Maucherat:
On Sun, Jan 27, 2019 at 2:18 AM Rainer Jung <rainer.j...@kippdata.de>
wrote:
When running the unit tests for trunk using Java 11 and APR connector I
run out of memory, first in threads, later in heap. The prpoblematic
tests are TestDefaultServletEncodingWithBom and
TestDefaultServletEncodingWithoutBom.
A thread dumps shows 540 sendfile threads like the following:
[junit] "http-apr-127.0.0.1-auto-540-Sendfile" #8113 daemon prio=5
There should be one sendfile thread (at most) per APR connector. 540 is
OK, then there's something wrong. The line above was an example. Later
examination on other Linux systems that are not resource restricted
showed more than 1000 of these sendfile threads during execution of
these two test classes close to the end of the respective test. I have a
thread dump with all numbers from 1 up to 1257 in one thread dump.
[junit] "http-apr-127.0.0.1-auto-1-Sendfile" #27 daemon prio=5
os_prio=0 cpu=0.21ms elapsed=146.90s tid=0x00007f112c7e5000 nid=0x7906
in Object.wait() [0x00007f10e42c0000]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(java.base@11.0.2/Native Method)
[junit] - waiting on <0x00000000c5548310> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Object.wait(java.base@11.0.2
/Object.java:328)
[junit] at
org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116)
[junit] - waiting to re-lock in wait() <0x00000000c5548310> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
[junit]
[junit] "http-apr-127.0.0.1-auto-2-Sendfile" #43 daemon prio=5
os_prio=0 cpu=0.10ms elapsed=146.20s tid=0x00007f112c62f000 nid=0x7916
in Object.wait() [0x00007f10e41bf000]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(java.base@11.0.2/Native Method)
[junit] - waiting on <0x00000000c5e0d590> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Object.wait(java.base@11.0.2
/Object.java:328)
[junit] at
org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116)
[junit] - waiting to re-lock in wait() <0x00000000c5e0d590> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
[junit]
[junit] "http-apr-127.0.0.1-auto-3-Sendfile" #58 daemon prio=5
os_prio=0 cpu=0.10ms elapsed=146.03s tid=0x00007f112c7e2800 nid=0x7925
in Object.wait() [0x00007f10e43c1000]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(java.base@11.0.2/Native Method)
[junit] - waiting on <0x00000000c5e0d748> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Object.wait(java.base@11.0.2
/Object.java:328)
[junit] at
org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116)
[junit] - waiting to re-lock in wait() <0x00000000c5e0d748> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
[junit]
...
[junit] "http-apr-127.0.0.1-auto-1257-Sendfile" #18868 daemon
prio=5 os_prio=0 cpu=0.08ms elapsed=0.23s tid=0x00007f112d015000
nid=0x4541 in Object.wait() [0x00007f107ca51000]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(java.base@11.0.2/Native Method)
[junit] - waiting on <no object reference available>
[junit] at java.lang.Object.wait(java.base@11.0.2
/Object.java:328)
[junit] at
org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116)
[junit] - waiting to re-lock in wait() <0x00000000cc7769c8> (a
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
[junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
[junit]
only a static counter to avoid running into a connector name conflict
when
the auto port is used. The sendfile thread is normally started and
stopped
with the connector. There's probably another cause for this apparent
resource leak (I don't get the error on my OpenJDK 11).
Did you try to get a thread dump while the test is executing? The errors
are due to resource restrictions on parts of my environment, but even on
the environment with no restrictions and errors I can see the huge
amount of threads (increasing over the time the test runs). For example
on RHEL 7 with Java 11 and tcnative 1.2.21.
The counter is a great indication of the lifecycle of the JVM.
Unfortunately also the threads with the lower counter values are all
still there.
Just pointing out the info about the counter. The sendfile stop didn't seem
correct, I can trace now that the sendfile run() is ending properly.
Thanks Rémy, that seems to have fixed it.
As a much smaller annoyance, now every single executing test writes one
new line:
WARNING [main] org.apache.tomcat.util.net.AprEndpoint$Sendfile.stop The
sendfile thread failed to stop in a timely manner
Waiting for the stop happens in a 50 iteration loop, each waiting 2
milliseconds, so 100ms in total. Since the system is slow, it could be,
that it takes so long, but I am astonished it takes that long always.
On the other hand the thread dumps show, that no old sendfile threads
are hanging around during the Bom tests. So finally they stop.
Thanks and regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
