If the distributedTest jobs are swapping then we have a bigger problem than just this test. We can crank down on the number of parallel jobs and see if that relieves the memory pressure, or we can check for machine types with more memory.
I can look at this on Monday, or maybe ping @Sean Goller <sgol...@pivotal.io> On Thu, Aug 22, 2019 at 9:35 AM Kirk Lund <kl...@apache.org> wrote: > So far Jens and I are unable to reproduce these test failures [1]. This > does not appear to be a test or java code problem. Based on google searches > [2] [3], the cause might be either: > > 1) the distributed test job is out of swap space when the Tomcat/Jetty > tests run > > 2) the /proc/sys/vm/overcommit_memory and /proc/sys/vm/overcommit_ratio may > be misconfigured > > I don't have sufficient concourse knowledge to attempt to fix or change > either of the above two settings in our precheckin environment. Can someone > please help me with this? > > The failure: > > > *OpenJDK 64-Bit Server VM warning: INFO: > os::commit_memory(0x00007f74a4ba4000, 65536, 1) failed; error='Not enough > space' (errno=12) [thread 26510 also had an error] * > > [1] https://concourse.apachegeode-ci.info/builds/88240 > [2] https://bugs.openjdk.java.net/browse/JDK-8216619 > [3] > > https://stackoverflow.com/questions/46801741/jvm-crashes-with-error-cannot-allocate-memory-errno-12 > > > On Wed, Aug 21, 2019 at 8:36 AM Jens Deppe <jde...@pivotal.io> wrote: > > > I can work with you on this if you're still blocked. > > > > --Jens > > > > On Tue, Aug 20, 2019 at 4:12 PM Kirk Lund <kl...@apache.org> wrote: > > > > > Does anyone know how to debug geode-assembly Jetty dunit tests that > fail > > to > > > launch modify_war? > > > > > > It passes 100% locally in intellij and with gradle cli. It only fails > in > > > concourse PR precheckin. > > > > > > Right now, this is the last thing blocking me from submitting a > non-draft > > > PR to move all log4j-core code from geode-core to geode-log4j. This is > > > blocking the creation of geode-log4j. > > > > > > The only changes in my branch are moving all log4j-core code from > > > geode-core to geode-log4j. > > > > > > If anyone else wants to see this change make it to develop, then I need > > > help! > > > > > > When the test tries to execute modify_war, it fails with the following > > > output and stack trace. No other info is available for debugging as > > > apparently this kills the gradle daemon. > > > > > > Failed PR precheckin dunit job: > > > https://concourse.apachegeode-ci.info/builds/88240 > > > PR: https://github.com/apache/geode/pull/3914 > > > > > > > Task :geode-assembly:distributedTest > > > > > > org.apache.geode.session.tests.Jetty9CachingClientServerTest > > > > containersShouldHavePersistentSessionData FAILED > > > java.lang.RuntimeException: Something very bad happened when trying > > to > > > start container > > > > > > > > > JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05_<unknown> > > > > > > Caused by: > > > java.lang.RuntimeException: Something very bad happened to this > > > container when starting. Check the cargo_logs folder for container > logs. > > > > > > Caused by: > > > java.io.IOException: Unable to run modify_war script, > > command: > > > > > > > > > [/tmp/geode_container_install17845041006471328987/cargo_modules/Apache_Geode_Modules-1.11.0-SNAPSHOT-AppServer/bin/modify_war, > > > -J, -Xmx2096m, -w, > > > > > > > > > /home/geode/geode/geode-assembly/build/distributedTest254/../../../extensions/session-testing-war/build/libs/session-testing-war.war, > > > -t, client-server, -o, > > > > > > > > > /tmp/geode_container_install17845041006471328987/cargo_wars/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f053692095078744488223.war, > > > -p, gemfire.cache.enable_local_cache=true, -p, > > > > > > > > > gemfire.property.log-file=/home/geode/geode/geode-assembly/build/distributedTest254/cargo_logs/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05/gemfire.log, > > > -p, > > > > > > > > > gemfire.property.cache-xml-file=/home/geode/geode/geode-assembly/build/distributedTest254/cargo_logs/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05/cache-client.xml] > > > log file: > > > ERROR: Error updating web.xml > > > ng: INFO: os::commit_memory(0x000000077d000000, 2147483648, > > 0) > > > failed; error='Not enough space' (errno=12) > > > > > >
