The failing WAN-related tests are still making up the majority of flaky tests<https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-mass-test-run/jobs/create-mass-test-run-report/builds/40> we are observing. It's a real issue, since the noise from flaky tests can easily hide very real bugs. That's why earlier this year several people spent a few months improving the reliability of our tests.
Mario, in July you said you were investigating this. Is that still ongoing? Do you need any help? ________________________________ From: Alexander Murmann <amurm...@apache.org> Sent: Tuesday, July 14, 2020 09:00 To: dev@geode.apache.org <dev@geode.apache.org> Cc: Alexander Murmann <amurm...@apache.org> Subject: Re: Odg: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes Thanks for looking into this, Mario! You are probably right, that the underlying issue might have been pre-existing and that the test is surfacing it. I am glad though that you are investigating, because a close to 30% fail rate is a problem. Something like this happens every once in a while and then someone has to do some work to resolve historical problems that they hadn't planned to addres. Thanks! On Tue, Jul 14, 2020 at 1:06 AM Mario Ivanac <mario.iva...@est.tech> wrote: > Hi, > > after adding additional checks in failing test, now I can see that test > are failing due to fault that some batch are distributed at stopping of GW > sender. > Cause of that, I suspect that this problem existed prior to this PR, but > this PR is first to introduce test to check this. > > I will continue to investigate this fault, but I can not locally reproduce > this fault, so this is slowing troubleshooting. > > BR, > Mario > ________________________________ > Šalje: Alexander Murmann <amurm...@apache.org> > Poslano: 14. srpnja 2020. 1:11 > Prima: Alexander Murmann <amurm...@apache.org> > Kopija: dev@geode.apache.org <dev@geode.apache.org>; Mario Ivanac > <mario.iva...@est.tech> > Predmet: Re: [INFO] Latest test run of 200 DistributedTestOpenJDK8 passes > > We continue to see these WAN tests adding a fail rate of just below 30% in > our mass test runs > < > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=STlN5I9iM8gRqsKndH6GKwuY5cVOskAgUjNgkoJvzCw%3D&reserved=0 > > > . > > That's a very significant fail rate that impacts our ability to get our > code committed with confidence. > > Can we resolve this issue? Otherwise, I think we need to consider reverting > GEODE-7458. > > On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <amurm...@apache.org> > wrote: > > > Looking more into this, it looks like this was introduced by the changes > > for GEODE-7458 - "Adding additional option in gfsh command "start gateway > > sender" to control clearing of existing queues". > > > > That happened about a month ago, but it's inherent to those flaky tests > > that we discover them only after a while. Nonetheless, they become paper > > cuts that ultimately slow us down substantially if they persist. > > > > @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the > > one making that change. Might you be able to take a look at making that > > test more reliable or reverting the change? > > > > Thank you! > > > > On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <amurm...@apache.org> > > wrote: > > > >> Thank you so much for sharing this, Mark! > >> > >> It looks like there is a big cluster around WAN Gateway. Is anyone > >> already looking into the WAN issues? > >> > >> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <hans...@vmware.com> > wrote: > >> > >>> FYI, the build success rate was around 90% or so about two months ago. > >>> > >>> Here are the DUnit tests that are currently failing in our tests, most > >>> likely in CI, and PR pipelines. > >>> > >>> Please let me know if you have any questions. > >>> > >>> Thanks, > >>> Mark > >>> > >>> > >>> > >>> > *********************************************************************************** > >>> > >>> Overall build success rate: 78.00000% (156 of 200) > >>> > >>> > >>> > *********************************************************************************** > >>> > >>> > >>> > >>> The following test methods see failures in more than one class. There > >>> may be a failing *TestBase class > >>> > >>> > >>> > >>> > *.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: > >>> 12 failures : > >>> > >>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures > >>> (96.000% success rate) > >>> > >>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures > >>> (98.000% success rate) > >>> > >>> > >>> > >>> > *.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived: > >>> 12 failures : > >>> > >>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5 > >>> failures (97.500% success rate) > >>> > >>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures > >>> (96.500% success rate) > >>> > >>> > >>> > >>> *.testPingWrongServer: 4 failures : > >>> > >>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate) > >>> > >>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate) > >>> > >>> > >>> > >>> > >>> > *********************************************************************************** > >>> > >>> > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest: > >>> 8 failures (96.000% success rate) > >>> > >>> > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=zcUAsAVVsUprEFeDpWryPa%2F4QTf0pQxcZ8yNpzFwvtQ%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=309OdZMeCRLDty3sTtCumrWt2T0b76JNA94EvwuB%2BSI%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=SWeCxshemmITfDqulXqu4gQVRPzMOXrgvf3LGcXSTUs%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=7gzxFbHVzKgpdZ1ZU8vhDYzXzU4cxyRPxgly6dIt31k%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=Xa8bjxPmysZluEn6HbvQzqbccJl%2B%2FsiGIhWYGeBP%2Ba4%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297231701&sdata=O%2B0CoquKmvf5eT%2Ff%2FMLoXVuojjwk4IeeyI5dIdJugYg%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=AILkL%2Fhz3I%2FasCXa8fkTSB9mA26gmDxJ4fe%2FeIiS27g%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=LF0Jr4d9jundS8cNJtWK%2FnRGVLBp4gsX8DUduuFbp0E%3D&reserved=0 > >>> > >>> > >>> > >>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures > >>> (99.000% success rate) > >>> > >>> > >>> > >>> testBucketCount > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=oYLB8lcafCMgeSVcwwuaIOdKzEzgg4ypKfS4HUrMHrU%3D&reserved=0 > >>> > >>> testBucketCount > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=O0jF6ARYoS0cIxzTNq%2BJQtHB30W45nrBj%2B5lOqu0esY%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest: > >>> 1 failures (99.500% success rate) > >>> > >>> > >>> > >>> testRegisterInterestAndMakePrimaryWithFullRedundancy > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=la5BbwTkvNS1gFlNf3yr%2BF21L0t2z6NY6Clgwv4WGr4%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest: > >>> 1 failures (99.500% success rate) > >>> > >>> > >>> > >>> testSimpleQueryOnLocator > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Li1jYIfu1biAC47UGz1zqJbAW6neNxgvcw1JaFeR2pk%3D&reserved=0 > >>> > >>> > >>> > >>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest: > >>> 1 failures (99.500% success rate) > >>> > >>> > >>> > >>> testPingWrongServer > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=u7%2FQXmha26M6yX4rjRspIRDOEwQZB0NjaCxwBBkfFws%3D&reserved=0 > >>> > >>> > >>> > >>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3 > >>> failures (98.500% success rate) > >>> > >>> > >>> > >>> > >>> > testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=jA3538gndW%2BM42LYlJBMzJX9K4psvoQiYxs0761Viog%3D&reserved=0 > >>> > >>> > >>> > testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=%2Fh6%2FPjO2f1nnZ7RzMoUnLmRM0Mq2XHlGlSblj7jlcPc%3D&reserved=0 > >>> > >>> testReplicatedSerialPropagationHAWithGroupTransactionEvents > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ikv%2FMsJVZbPhGRndMcviXfSUsYjMqob2NAZuW4GbfMw%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest: > >>> 1 failures (99.500% success rate) > >>> > >>> > >>> > >>> testDistributedDeadlockWithDLock > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=ZMgFxv%2FpfiVF4xSl5C0VtJSfCVoeA5phFRJF8IS79Lk%3D&reserved=0 > >>> > >>> > >>> > >>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500% > >>> success rate) > >>> > >>> > >>> > >>> testStartTwoLocatorsWithMultiKeystoreSSL > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=Ro4ej1P%2BTLAN%2FM%2B8v9DVFbK%2B8cSDtCKdjeoYEIfbWgw%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: > >>> 5 failures (97.500% success rate) > >>> > >>> > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=TAWaGwXo4t3%2FCQp3MewJKEn7p1rdQgp%2FVDWHqQxva4c%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297241696&sdata=lApVu2gdSk38MjHjZlWZhPWyQ5fOez2Sma5WmYcNE38%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=6gC%2BqQusVFP5JqeHkxtSvk8%2FWCRIEg90ZeiWomJ1NrE%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest: > >>> 1 failures (99.500% success rate) > >>> > >>> > >>> > >>> > >>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=rrg9pcWvSZgJ106rjbsS8wpCBhX20er1hdNhW0qGrZ4%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest: > >>> 7 failures (96.500% success rate) > >>> > >>> > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=7R6DvYToK1cwdTfBK93HA9y83qw9OKd4Kj%2BbEe%2F8Yd4%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=1RXrV5Zw0kQSLDrPLu45DlWy4ZR%2B2hs7rfyk8BAFeG4%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=DJ9zA4EJyrjd01eidiCznYn%2FOAVtVfzYzxsfapCGUD4%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=vF7zJq%2BZnOIa1W6oVrfKRJlV6nFIBitzjCVPPCgc6CQ%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=%2FvRCNF0K8V9LfYTsdDxRniadtdXVuV7cZZZsmQTydw0%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=WDNL8hapq0GmB%2BrxSxFYJ0c4nArV8k15m79OkwrtASY%3D&reserved=0 > >>> > >>> > >>> > testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=8%2B%2FBMm9eeXPcjAL4Hj94aduYQUyDrJVgPqB5%2FSqZ4kY%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest: > >>> 4 failures (98.000% success rate) > >>> > >>> > >>> > >>> testMultipleColocatedChildPRsMissingWithSequencedStart > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=l7RlB5fYmzwGJvYuy5QFIPWJd5mi1o5BAlCYff0XeDQ%3D&reserved=0 > >>> > >>> testMissingColocatedChildPRDueToDelayedStart > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=SgLx2XahBD8x2Cez3aK%2BhShvVc%2BP3SFmMfeqB3pdHLM%3D&reserved=0 > >>> > >>> testHierarchyOfColocatedChildPRsMissingGrandchild > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=3Y5qtB5P6Gz8ey%2F6eeEOOGdLptT6LVIBVWLpF4lrRxs%3D&reserved=0 > >>> > >>> testHierarchyOfColocatedChildPRsMissingGrandchild > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297251688&sdata=n6qxZzVKFYqiTY7kUw%2BoQfZKE2aUvbBXs6cXAcqBZaw%3D&reserved=0 > >>> > >>> > >>> > >>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures > >>> (99.000% success rate) > >>> > >>> > >>> > >>> testGroupsInAllVMs > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=prw%2BEBBIgOH8BWwD5SBpvVd%2BvdQSh%2FOJf8Xk6zPZ7yk%3D&reserved=0 > >>> > >>> testGroupsInAllVMs > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=g2PQp%2FiMZEVjc4zSOxNhks48UTPdEAVGyhdJQueipuQ%3D&reserved=0 > >>> > >>> > >>> > >>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures > >>> (97.500% success rate) > >>> > >>> > >>> > >>> serverMXBeansOnServerAreUnaffectedByLocatorCrash > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=zVXQeZRlv6qJGvlpOFvJIt5WDzvSAq2ohyQ1DRZ1xWE%3D&reserved=0 > >>> > >>> > >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=t5BIGh1%2BcB6SKdUviko%2FghBDtJIrmEUVMGQVXdEhHbA%3D&reserved=0 > >>> > >>> > >>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GyvOWUusGUi48vS6wWA6GrJN3J5aAMbqeXxy2VevLXU%3D&reserved=0 > >>> > >>> locatorHasMemberTypeMXBeansForBothLocators > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=N261EBctQXEW0FR0XTTHd4w6nmrEkj%2F%2FLd8e14m0cnw%3D&reserved=0 > >>> > >>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=QNtL7hTrfxZHkLfKGx7P01m4quS6%2FhkkjyYdhA7J8bA%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: > >>> 4 failures (98.000% success rate) > >>> > >>> > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=TPAyTUGYWUv%2FifU58XondaeGNJBXmB8sskEp0J6R8l4%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=gVTwnIWWDDjOhkfyM4SzYegMuQjyaXDg7sFkzkhUyJA%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=iErMnLzSVjZpkZO%2FJ%2BETC0QbO0neLHNlA41OkXWWBuk%3D&reserved=0 > >>> > >>> > >>> > testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived > >>> > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=GptTVqXhyQlHTByVbEoQJtjF%2FWc4xlC2KLHHZ3qhbrw%3D&reserved=0 > >>> > >>> > >>> > >>> > org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest: > >>> 3 failures (98.500% success rate) > >>> > >>> > >>> > >>> testPingWrongServer > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297261689&sdata=S9EbS96H9rYgoCfTtXXTVbcMZAIBZ%2BlIbQgElXKzX98%3D&reserved=0 > >>> > >>> testPingWrongServer > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=4OQQFRPSwtx3QcENDyVCUsoMJd0K%2Bwl5o759cyZfsYw%3D&reserved=0 > >>> > >>> testPingWrongServer > >>> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=02%7C01%7Camurmann%40vmware.com%7C3eab11cb15d84879960f08d8280f073b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637303392297271685&sdata=XjIc4NZPwT4HlRnEL9zfWcB5AlxLiO%2B9C9jf77TJ0Dg%3D&reserved=0 > >>> > >>> >
