Actually, it looks like the problem is that we are *not* using docker containers for the acceptance tests. Check this out, in gradle/docker.gradle. Since acceptance tests use the default port, this means the test are guaranteed to be flaky, especially since we are running them in parallel:
// ACCEPTANCE TEST NEEDS DOCKER-COMPOSE TO WORK WITHIN DOCKER FIRST // acceptanceTest.configure(dockerConfig) I'm not sure what changed that is causing the tests to fail more often now, but maybe a test ordering change? -Dan On Tue, Aug 21, 2018 at 10:52 AM, Kenneth Howe <kh...@pivotal.io> wrote: > > > > On Aug 21, 2018, at 10:44 AM, Kirk Lund <kl...@apache.org> wrote: > > > > GEODE-5590 would seem to imply that GfshRule does not have an adequate > safe > > guard? If it spawns a server process which binds to the default server > port > > and that process persists after the test then we need better tearDown. > > > Yes, that does appear to be the case. The current failures are apparently > due to incomplete > teardown between tests within a test class. > > I am attempting to reproduce the failures on a consistent basis for > debugging the problem. > > > > Actually I thought we were using Docker to run each AcceptanceTest in > > isolation. Then when the test finishes the Docker instances goes away. > Did > > we stop using Docker for these? > > > > On Tue, Aug 21, 2018 at 10:25 AM, Sai Boorlagadda < > sai.boorlaga...@gmail.com > >> wrote: > > > >> DeployWithLargeJarTest & PutCommandWithJsonTest are flaky on Develop. > >> > >> DeployWithLargeJarTest - > >> https://concourse.apachegeode-ci.info/teams/main/pipelines/ > >> develop/jobs/AcceptanceTest/builds/335 > >> PutCommandWithJsonTest - > >> https://concourse.apachegeode-ci.info/teams/main/pipelines/ > >> develop/jobs/AcceptanceTest/builds/334 > >> > >> On Tue, Aug 21, 2018 at 10:18 AM Sai Boorlagadda < > >> sai.boorlaga...@gmail.com> > >> wrote: > >> > >>> The metrics job themselves will be green (as they complete to success) > >> but > >>> you can expand the get_metrics task output and see that build#20 > started > >>> reporting these failures, so probably these are due to recent changes > on > >>> develop. I believe these metrics are from develop CI test runs. > >>> > >>> On Tue, Aug 21, 2018 at 10:15 AM Kirk Lund <kl...@apache.org> wrote: > >>> > >>>> Those metrics show AcceptanceTests consistently GREEN. Do these > metrics > >>>> include test failures from pull request precheckin runs like mine? Or > >> does > >>>> it just cover CI test runs? > >>>> > >>>> On Tue, Aug 21, 2018 at 10:09 AM, Sai Boorlagadda < > >>>> sai.boorlaga...@gmail.com > >>>>> wrote: > >>>> > >>>>> Metrics show these started failing recently. > >>>>> > >>>>> > >>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/ > >> metrics/jobs/ > >>>>> GeodeAcceptanceTestMetrics/builds/20 > >>>>> > >>>>> On Tue, Aug 21, 2018 at 10:07 AM Kirk Lund <kl...@apache.org> wrote: > >>>>> > >>>>>> Are PutCommandWithJsonTest and DeployWithLargeJarTest known to be > >>>> flaky? > >>>>>> > >>>>>> My latest pull request failed with these two failures and all I did > >>>> was > >>>>>> extract LocalRegion.validateRegionName and improve unit testing of > >>>>>> RegionNameValidation. No other tests failed for me. > >>>>>> > >>>>>>> Task :geode-assembly:acceptanceTest > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:619> > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:620> > >>>>>> > >>>> org.apache.geode.management.internal.cli.commands. > >> PutCommandWithJsonTest > >>>>>>> putWithJsonString FAILED > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:621> > >>>>>> org.junit.ComparisonFailure: expected:<[0]> but was:<[1]> > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:622> > >>>>>> at > >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >>>>>> Method) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:623> > >>>>>> at > >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance( > >>>>> NativeConstructorAccessorImpl.java:62) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:624> > >>>>>> at > >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > >>>>> DelegatingConstructorAccessorImpl.java:45) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:625> > >>>>>> at > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript. > >>>>> awaitIfNecessary(GfshScript.java:117) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:626> > >>>>>> at > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshRule.execute( > >>>>> GfshRule.java:135) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:627> > >>>>>> at > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript.execute( > >>>>> GfshScript.java:106) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:628> > >>>>>> at > >>>>>> org.apache.geode.management.internal.cli.commands. > >>>>> PutCommandWithJsonTest.putWithJsonString( > PutCommandWithJsonTest.java: > >> 55) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:629> > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:630> > >>>>>> > >>>> org.apache.geode.management.internal.cli.commands. > >> DeployWithLargeJarTest > >>>>>>> deployLargeSetOfJars FAILED > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:631> > >>>>>> org.junit.ComparisonFailure: expected:<[0]> but was:<[1]> > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:632> > >>>>>> at > >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >>>>>> Method) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:633> > >>>>>> at > >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance( > >>>>> NativeConstructorAccessorImpl.java:62) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:634> > >>>>>> at > >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > >>>>> DelegatingConstructorAccessorImpl.java:45) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:635> > >>>>>> at > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript. > >>>>> awaitIfNecessary(GfshScript.java:117) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:636> > >>>>>> at > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshRule.execute( > >>>>> GfshRule.java:135) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:637> > >>>>>> at > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript.execute( > >>>>> GfshScript.java:106) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:638> > >>>>>> at > >>>>>> org.apache.geode.management.internal.cli.commands. > >>>>> DeployWithLargeJarTest.deployLargeSetOfJars( > >> DeployWithLargeJarTest.java: > >>>>> 41) > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:639> > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680#L5b60bc1a:640> > >>>>>>> Task :geode-assembly:acceptanceTest FAILED > >>>>>> > >>>>> > >>>> > >>> > >> > >
