Oops... I mean "orphaned processes" On Tue, Aug 21, 2018 at 11:13 AM, Kirk Lund <kl...@apache.org> wrote:
> So, we need to disable running them in parallel and improve the tearDown > to make sure these tests don't leave an orphaned tests behind. > > On Tue, Aug 21, 2018 at 11:11 AM, Sai Boorlagadda < > sai.boorlaga...@gmail.com> wrote: > >> jdbc-connector acceptance tests need docker-in-docker (also >> docker-compose) >> to spin up mysql and postgres. >> >> On Tue, Aug 21, 2018 at 11:04 AM Dan Smith <dsm...@pivotal.io> wrote: >> >> > Actually, it looks like the problem is that we are *not* using docker >> > containers for the acceptance tests. Check this out, in >> > gradle/docker.gradle. Since acceptance tests use the default port, this >> > means the test are guaranteed to be flaky, especially since we are >> running >> > them in parallel: >> > >> > // ACCEPTANCE TEST NEEDS DOCKER-COMPOSE TO WORK WITHIN DOCKER FIRST >> > // acceptanceTest.configure(dockerConfig) >> > >> > I'm not sure what changed that is causing the tests to fail more often >> now, >> > but maybe a test ordering change? >> > >> > -Dan >> > >> > >> > >> > On Tue, Aug 21, 2018 at 10:52 AM, Kenneth Howe <kh...@pivotal.io> >> wrote: >> > >> > > >> > > >> > > > On Aug 21, 2018, at 10:44 AM, Kirk Lund <kl...@apache.org> wrote: >> > > > >> > > > GEODE-5590 would seem to imply that GfshRule does not have an >> adequate >> > > safe >> > > > guard? If it spawns a server process which binds to the default >> server >> > > port >> > > > and that process persists after the test then we need better >> tearDown. >> > > > >> > > Yes, that does appear to be the case. The current failures are >> apparently >> > > due to incomplete >> > > teardown between tests within a test class. >> > > >> > > I am attempting to reproduce the failures on a consistent basis for >> > > debugging the problem. >> > > >> > > >> > > > Actually I thought we were using Docker to run each AcceptanceTest >> in >> > > > isolation. Then when the test finishes the Docker instances goes >> away. >> > > Did >> > > > we stop using Docker for these? >> > > > >> > > > On Tue, Aug 21, 2018 at 10:25 AM, Sai Boorlagadda < >> > > sai.boorlaga...@gmail.com >> > > >> wrote: >> > > > >> > > >> DeployWithLargeJarTest & PutCommandWithJsonTest are flaky on >> Develop. >> > > >> >> > > >> DeployWithLargeJarTest - >> > > >> https://concourse.apachegeode-ci.info/teams/main/pipelines/ >> > > >> develop/jobs/AcceptanceTest/builds/335 >> > > >> PutCommandWithJsonTest - >> > > >> https://concourse.apachegeode-ci.info/teams/main/pipelines/ >> > > >> develop/jobs/AcceptanceTest/builds/334 >> > > >> >> > > >> On Tue, Aug 21, 2018 at 10:18 AM Sai Boorlagadda < >> > > >> sai.boorlaga...@gmail.com> >> > > >> wrote: >> > > >> >> > > >>> The metrics job themselves will be green (as they complete to >> > success) >> > > >> but >> > > >>> you can expand the get_metrics task output and see that build#20 >> > > started >> > > >>> reporting these failures, so probably these are due to recent >> changes >> > > on >> > > >>> develop. I believe these metrics are from develop CI test runs. >> > > >>> >> > > >>> On Tue, Aug 21, 2018 at 10:15 AM Kirk Lund <kl...@apache.org> >> wrote: >> > > >>> >> > > >>>> Those metrics show AcceptanceTests consistently GREEN. Do these >> > > metrics >> > > >>>> include test failures from pull request precheckin runs like >> mine? >> > Or >> > > >> does >> > > >>>> it just cover CI test runs? >> > > >>>> >> > > >>>> On Tue, Aug 21, 2018 at 10:09 AM, Sai Boorlagadda < >> > > >>>> sai.boorlaga...@gmail.com >> > > >>>>> wrote: >> > > >>>> >> > > >>>>> Metrics show these started failing recently. >> > > >>>>> >> > > >>>>> >> > > >>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/ >> > > >> metrics/jobs/ >> > > >>>>> GeodeAcceptanceTestMetrics/builds/20 >> > > >>>>> >> > > >>>>> On Tue, Aug 21, 2018 at 10:07 AM Kirk Lund <kl...@apache.org> >> > wrote: >> > > >>>>> >> > > >>>>>> Are PutCommandWithJsonTest and DeployWithLargeJarTest known to >> be >> > > >>>> flaky? >> > > >>>>>> >> > > >>>>>> My latest pull request failed with these two failures and all I >> > did >> > > >>>> was >> > > >>>>>> extract LocalRegion.validateRegionName and improve unit >> testing of >> > > >>>>>> RegionNameValidation. No other tests failed for me. >> > > >>>>>> >> > > >>>>>>> Task :geode-assembly:acceptanceTest >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:619 >> > > >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:620 >> > > >> > > >>>>>> >> > > >>>> org.apache.geode.management.internal.cli.commands. >> > > >> PutCommandWithJsonTest >> > > >>>>>>> putWithJsonString FAILED >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:621 >> > > >> > > >>>>>> org.junit.ComparisonFailure: expected:<[0]> but was:<[1]> >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:622 >> > > >> > > >>>>>> at >> > > >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> > > >>>>>> Method) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:623 >> > > >> > > >>>>>> at >> > > >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance( >> > > >>>>> NativeConstructorAccessorImpl.java:62) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:624 >> > > >> > > >>>>>> at >> > > >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance( >> > > >>>>> DelegatingConstructorAccessorImpl.java:45) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:625 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript. >> > > >>>>> awaitIfNecessary(GfshScript.java:117) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:626 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshRule.execute( >> > > >>>>> GfshRule.java:135) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:627 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript.execute( >> > > >>>>> GfshScript.java:106) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:628 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.management.internal.cli.commands. >> > > >>>>> PutCommandWithJsonTest.putWithJsonString( >> > > PutCommandWithJsonTest.java: >> > > >> 55) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:629 >> > > >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:630 >> > > >> > > >>>>>> >> > > >>>> org.apache.geode.management.internal.cli.commands. >> > > >> DeployWithLargeJarTest >> > > >>>>>>> deployLargeSetOfJars FAILED >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:631 >> > > >> > > >>>>>> org.junit.ComparisonFailure: expected:<[0]> but was:<[1]> >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:632 >> > > >> > > >>>>>> at >> > > >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> > > >>>>>> Method) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:633 >> > > >> > > >>>>>> at >> > > >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance( >> > > >>>>> NativeConstructorAccessorImpl.java:62) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:634 >> > > >> > > >>>>>> at >> > > >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance( >> > > >>>>> DelegatingConstructorAccessorImpl.java:45) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:635 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript. >> > > >>>>> awaitIfNecessary(GfshScript.java:117) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:636 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshRule.execute( >> > > >>>>> GfshRule.java:135) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:637 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript.execute( >> > > >>>>> GfshScript.java:106) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:638 >> > > >> > > >>>>>> at >> > > >>>>>> org.apache.geode.management.internal.cli.commands. >> > > >>>>> DeployWithLargeJarTest.deployLargeSetOfJars( >> > > >> DeployWithLargeJarTest.java: >> > > >>>>> 41) >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:639 >> > > >> > > >>>>>> <https://concourse.apachegeode-ci.info/builds/19680# >> L5b60bc1a:640 >> > > >> > > >>>>>>> Task :geode-assembly:acceptanceTest FAILED >> > > >>>>>> >> > > >>>>> >> > > >>>> >> > > >>> >> > > >> >> > > >> > > >> > >> > >
