The jobs just launch heavy lifter linux instances to run Gradle so this is just a pretty good sized plain old linux host. Sounds like maybe there are too many integration tests running in parallel eating up more memory than before. This may cause the virtual memory manager to panic and start killing some processes to preserve the machine health. The instance may need more ram or we need to reduce the parallel tests running.
-Jake > On Aug 22, 2019, at 1:10 PM, Kirk Lund <kl...@pivotal.io> wrote: > > I have multiple PRs submitted from multiple branches, all rebased etc. They > all fail and the gradle process dies. The failure message (when there is > one) always says the same thing... > > Some sort of environment issue is causing DistributedTestOpenJDK11 to fail > [1] and the Gradle daemon to die (on every PR involving the creation of > geode-log4j and moving the log4j core classes to that module). > > Based on google searches [2] [3], the cause might be either: > > 1. > > the distributed test job is out of swap space when the Tomcat/Jetty > tests run > 2. > > the /proc/sys/vm/overcommit_memory and /proc/sys/vm/overcommit_ratio may > be misconfigured > > I don't have sufficient concourse knowledge to attempt to fix or change > either of the above two settings in our precheckin environment. Can someone > please help me with this? > > The failure: > > OpenJDK 64-Bit Server VM warning: INFO: > os::commit_memory(0x00007f74a4ba4000, 65536, 1) failed; error='Not > enough space' (errno=12) > [thread 26510 also had an error] > > [1] https://concourse.apachegeode-ci.info/builds/88240 > [2] https://bugs.openjdk.java.net/browse/JDK-8216619 > [3] > https://stackoverflow.com/questions/46801741/jvm-crashes-with-error-cannot-allocate-memory-errno-12 > > On Thu, Aug 22, 2019 at 11:16 AM Jacob Barrett <jbarr...@pivotal.io> wrote: > >> I wonder if the auto merge creates something funky. Try rebasing or >> merging your branch first then push. >> >>> On Aug 22, 2019, at 11:11 AM, Bruce Schuchardt <bschucha...@pivotal.io> >> wrote: >>> >>> Using --no-daemon doesn't stop it from launching a daemon. It just >> tells it to shut down the daemon when the build is done. >>> >>>> On 8/22/19 11:05 AM, Michael Oleske wrote: >>>> Not sure what the exact issue in this particular case, but I suspect we >>>> might not want to run with the Gradle daemon in CI. The Gradle daemon >> is >>>> really meant to help local building (see >>>> https://docs.gradle.org/current/userguide/gradle_daemon.html). That >> link >>>> explicitly calls out that you may not want to use the Gradle daemon in >> CI, >>>> so we could use the --no-daemon flag to disable the Gradle daemon in CI >> if >>>> we feel that would help eliminate extra problems. >>>> >>>> -michael >>>> >>>>> On Thu, Aug 22, 2019 at 10:36 AM Kirk Lund <kl...@apache.org> wrote: >>>>> >>>>> Can someone please look at >>>>> https://concourse.apachegeode-ci.info/builds/88869 and help me >> determine >>>>> what happened to the Gradle daemon? >>>>> >>>>>> Task :geode-assembly:distributedTest >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:714> >>>>>> Task :geode-core:distributedTest >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:715> >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:716> >>>>> The message received from the daemon indicates that the daemon has >>>>> disappeared. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:717> >>>>> Build request sent: Build{id=0311dae5-5824-4303-acfe-ceb50507a6a4, >>>>> currentDir=/home/geode/geode} >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:718> >>>>> Attempting to read last messages from the daemon log... >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:719> >>>>> Daemon pid: 5950 >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:720> >>>>> log file: /home/geode/.gradle/daemon/5.4/daemon-5950.out.log >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:721> >>>>> ----- Last 20 lines from daemon log file - daemon-5950.out.log ----- >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:722> >>>>> Note: Some input files use or override a deprecated API. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:723> >>>>> Note: Recompile with -Xlint:deprecation for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:724> >>>>> Note: Some input files use unchecked or unsafe operations. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:725> >>>>> Note: Recompile with -Xlint:unchecked for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:726> >>>>> Note: Some input files use or override a deprecated API. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:727> >>>>> Note: Recompile with -Xlint:deprecation for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:728> >>>>> Note: Some input files use unchecked or unsafe operations. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:729> >>>>> Note: Recompile with -Xlint:unchecked for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:730> >>>>> Note: Some input files use or override a deprecated API. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:731> >>>>> Note: Recompile with -Xlint:deprecation for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:732> >>>>> Note: Some input files use unchecked or unsafe operations. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:733> >>>>> Note: Recompile with -Xlint:unchecked for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:734> >>>>> Note: Some input files use or override a deprecated API. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:735> >>>>> Note: Recompile with -Xlint:deprecation for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:736> >>>>> Note: Some input files use unchecked or unsafe operations. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:737> >>>>> Note: Recompile with -Xlint:unchecked for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:738> >>>>> Note: Some input files use or override a deprecated API. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:739> >>>>> Note: Recompile with -Xlint:deprecation for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:740> >>>>> Note: Some input files use unchecked or unsafe operations. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:741> >>>>> Note: Recompile with -Xlint:unchecked for details. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:742> >>>>> ----- End of the daemon log ----- >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:743> >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:744> >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:745> >>>>> FAILURE: Build failed with an exception. >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:746> >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:747> >>>>> * What went wrong: >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:748> >>>>> Gradle build daemon disappeared unexpectedly (it may have been killed >>>>> or may have crashed) >>>>> <https://concourse.apachegeode-ci.info/builds/88869#L5d4c28d5:749> >>>>> >>
