On Mon, Jan 28, 2019 at 11:16 AM Rainer Jung <rainer.j...@kippdata.de> wrote:
> Am 28.01.2019 um 10:50 schrieb Rémy Maucherat: > > On Sun, Jan 27, 2019 at 2:18 AM Rainer Jung <rainer.j...@kippdata.de> > wrote: > > > >> When running the unit tests for trunk using Java 11 and APR connector I > >> run out of memory, first in threads, later in heap. The prpoblematic > >> tests are TestDefaultServletEncodingWithBom and > >> TestDefaultServletEncodingWithoutBom. > >> > >> A thread dumps shows 540 sendfile threads like the following: > >> > >> [junit] "http-apr-127.0.0.1-auto-540-Sendfile" #8113 daemon prio=5 > >> > > > > There should be one sendfile thread (at most) per APR connector. 540 is > > OK, then there's something wrong. The line above was an example. Later > examination on other Linux systems that are not resource restricted > showed more than 1000 of these sendfile threads during execution of > these two test classes close to the end of the respective test. I have a > thread dump with all numbers from 1 up to 1257 in one thread dump. > > [junit] "http-apr-127.0.0.1-auto-1-Sendfile" #27 daemon prio=5 > os_prio=0 cpu=0.21ms elapsed=146.90s tid=0x00007f112c7e5000 nid=0x7906 > in Object.wait() [0x00007f10e42c0000] > [junit] java.lang.Thread.State: WAITING (on object monitor) > [junit] at java.lang.Object.wait(java.base@11.0.2/Native Method) > [junit] - waiting on <0x00000000c5548310> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Object.wait(java.base@11.0.2 > /Object.java:328) > [junit] at > org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116) > [junit] - waiting to re-lock in wait() <0x00000000c5548310> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834) > [junit] > [junit] "http-apr-127.0.0.1-auto-2-Sendfile" #43 daemon prio=5 > os_prio=0 cpu=0.10ms elapsed=146.20s tid=0x00007f112c62f000 nid=0x7916 > in Object.wait() [0x00007f10e41bf000] > [junit] java.lang.Thread.State: WAITING (on object monitor) > [junit] at java.lang.Object.wait(java.base@11.0.2/Native Method) > [junit] - waiting on <0x00000000c5e0d590> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Object.wait(java.base@11.0.2 > /Object.java:328) > [junit] at > org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116) > [junit] - waiting to re-lock in wait() <0x00000000c5e0d590> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834) > [junit] > [junit] "http-apr-127.0.0.1-auto-3-Sendfile" #58 daemon prio=5 > os_prio=0 cpu=0.10ms elapsed=146.03s tid=0x00007f112c7e2800 nid=0x7925 > in Object.wait() [0x00007f10e43c1000] > [junit] java.lang.Thread.State: WAITING (on object monitor) > [junit] at java.lang.Object.wait(java.base@11.0.2/Native Method) > [junit] - waiting on <0x00000000c5e0d748> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Object.wait(java.base@11.0.2 > /Object.java:328) > [junit] at > org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116) > [junit] - waiting to re-lock in wait() <0x00000000c5e0d748> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834) > [junit] > ... > [junit] "http-apr-127.0.0.1-auto-1257-Sendfile" #18868 daemon > prio=5 os_prio=0 cpu=0.08ms elapsed=0.23s tid=0x00007f112d015000 > nid=0x4541 in Object.wait() [0x00007f107ca51000] > [junit] java.lang.Thread.State: WAITING (on object monitor) > [junit] at java.lang.Object.wait(java.base@11.0.2/Native Method) > [junit] - waiting on <no object reference available> > [junit] at java.lang.Object.wait(java.base@11.0.2 > /Object.java:328) > [junit] at > org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2116) > [junit] - waiting to re-lock in wait() <0x00000000cc7769c8> (a > org.apache.tomcat.util.net.AprEndpoint$Sendfile) > [junit] at java.lang.Thread.run(java.base@11.0.2/Thread.java:834) > [junit] > > > only a static counter to avoid running into a connector name conflict > when > > the auto port is used. The sendfile thread is normally started and > stopped > > with the connector. There's probably another cause for this apparent > > resource leak (I don't get the error on my OpenJDK 11). > > Did you try to get a thread dump while the test is executing? The errors > are due to resource restrictions on parts of my environment, but even on > the environment with no restrictions and errors I can see the huge > amount of threads (increasing over the time the test runs). For example > on RHEL 7 with Java 11 and tcnative 1.2.21. > > > The counter is a great indication of the lifecycle of the JVM. > > Unfortunately also the threads with the lower counter values are all > still there. > Just pointing out the info about the counter. The sendfile stop didn't seem correct, I can trace now that the sendfile run() is ending properly. Rémy > > Regards, > > Rainer > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
