The previous statement about 30% didn't make sense to me, so I thought I would
throw in this tidbit...
70 failures out 75 failures in 200 runs are caused by WAN.
Thanks,
Mark
On 7/13/20, 4:11 PM, "Alexander Murmann" <amurm...@apache.org> wrote:
We continue to see these WAN tests adding a fail rate of just below 30% in
our mass test runs
<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-develop-mass-test-run%2Fjobs%2Fcreate-mass-test-run-report%2Fbuilds%2F10&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=PZs6%2B%2F2vRL94anXVe0h6xt7U%2BwGR%2BKPbu7ZHLMr4vGQ%3D&reserved=0>
.
That's a very significant fail rate that impacts our ability to get our
code committed with confidence.
Can we resolve this issue? Otherwise, I think we need to consider reverting
GEODE-7458.
On Fri, Jun 19, 2020 at 3:28 PM Alexander Murmann <amurm...@apache.org>
wrote:
> Looking more into this, it looks like this was introduced by the changes
> for GEODE-7458 - "Adding additional option in gfsh command "start gateway
> sender" to control clearing of existing queues".
>
> That happened about a month ago, but it's inherent to those flaky tests
> that we discover them only after a while. Nonetheless, they become paper
> cuts that ultimately slow us down substantially if they persist.
>
> @Mario Ivanac If I am correct and GEODE-7458 introduced this you were the
> one making that change. Might you be able to take a look at making that
> test more reliable or reverting the change?
>
> Thank you!
>
> On Fri, Jun 19, 2020 at 7:57 AM Alexander Murmann <amurm...@apache.org>
> wrote:
>
>> Thank you so much for sharing this, Mark!
>>
>> It looks like there is a big cluster around WAN Gateway. Is anyone
>> already looking into the WAN issues?
>>
>> On Thu, Jun 18, 2020 at 10:06 PM Mark Hanson <hans...@vmware.com> wrote:
>>
>>> FYI, the build success rate was around 90% or so about two months ago.
>>>
>>> Here are the DUnit tests that are currently failing in our tests, most
>>> likely in CI, and PR pipelines.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>>
***********************************************************************************
>>>
>>> Overall build success rate: 78.00000% (156 of 200)
>>>
>>>
>>>
***********************************************************************************
>>>
>>>
>>>
>>> The following test methods see failures in more than one class. There
>>> may be a failing *TestBase class
>>>
>>>
>>>
>>>
*.testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> SerialWANPersistenceEnabledGatewaySenderDUnitTest: 8 failures
>>> (96.000% success rate)
>>>
>>> SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 4 failures
>>> (98.000% success rate)
>>>
>>>
>>>
>>>
*.testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived:
>>> 12 failures :
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest: 5
>>> failures (97.500% success rate)
>>>
>>> ParallelWANPersistenceEnabledGatewaySenderDUnitTest: 7 failures
>>> (96.500% success rate)
>>>
>>>
>>>
>>> *.testPingWrongServer: 4 failures :
>>>
>>> ClientServerMiscSelectorDUnitTest: 3 failures (98.500% success rate)
>>>
>>> ClientServerMiscDUnitTest: 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>>
***********************************************************************************
>>>
>>>
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.wan.serial.SerialWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 8 failures (96.000% success rate)
>>>
>>>
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3539&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=ILB4NvS0zakenbq00pfHjOZlb9pt7n60%2FpjNUV%2FUQp8%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3526&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=4pejQ84CaPxlR6terHjOb4MlSDhqmiZWw5FMfshFbL0%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3505&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835802218&sdata=0sOJRUud9GWXHo3iPgSU53z9Eyi9sh7LiYSXSRYCg38%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3435&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=iJJyU6KNfWTGF3cSdgfQzCtbh%2BroLJxhVfW0HL4NVKk%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3414&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=Qd90VpFi1ckXArWAerACT8QKQi5S3DPF9O%2Fc%2FWuOras%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3391&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=sybHZb4TXpDQndMWJazxYqpC8YL2yLCKcinJomP6WP0%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3363&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=AlBfHXFJCbKYhqOrO3c7n0vFWBK3n5jNmJt3ecQFS8Q%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3359&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=0Kj7PBqxzd8M53Hh7F2r%2B9mtYj57Gfc3WxCOVPCTYVo%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.management.MemberMXBeanDistributedTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testBucketCount
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=rE%2FYuEs6HBgpz0P3WYbe%2Bt3goceTlR%2Bs51tqzhDkFsA%3D&reserved=0
>>>
>>> testBucketCount
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3411&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=7M7j%2BVhhYDA9QpXQWfCr9nXi9Krl1HMObVJk8D%2FjufY%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.tier.sockets.RedundancyLevelPart3DUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testRegisterInterestAndMakePrimaryWithFullRedundancy
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3381&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=J0lyeA9l1b0gT4A0PR4N5KKMBySOQrg8zWCgKptE7iQ%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.management.internal.cli.commands.QueryCommandOverHttpDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testSimpleQueryOnLocator
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3516&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=FARo2AYcibOpGuFwt25N7s8rfUDddLe5QkhR4OxFJ8k%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.tier.sockets.ClientServerMiscDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3400&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=ORdmb4v2Lrp2SxI8HtiToZN%2BvSSddgg4OqzFuexvrSo%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.internal.cache.wan.serial.SerialWANStatsDUnitTest: 3
>>> failures (98.500% success rate)
>>>
>>>
>>>
>>>
>>>
testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3495&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=G%2Bxs28AKlk%2Fa9PjMg0X0xP8tHzfBpa8IWnqCUuzNdik%3D&reserved=0
>>>
>>>
>>>
testReplicatedSerialPropagationWithGroupTransactionEventsSendsBatchesWithCompleteTransactions
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3470&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=GAr47n034NLpEAgyYMM0%2Bg%2BcJIX5wV%2BAAAnKef3OEj0%3D&reserved=0
>>>
>>> testReplicatedSerialPropagationHAWithGroupTransactionEvents
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3479&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=z81ZJBzMqT%2F4SR3DySZgzO7GA8EnOY4jgNsqyUi6d4o%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.distributed.internal.deadlock.GemFireDeadlockDetectorDUnitTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>> testDistributedDeadlockWithDLock
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3421&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835812214&sdata=Ux54TOh1LLp9XRAjdbMK0scUn70V32Up%2FV78RjTtr8A%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.distributed.LocatorDUnitTest: 1 failures (99.500%
>>> success rate)
>>>
>>>
>>>
>>> testStartTwoLocatorsWithMultiKeystoreSSL
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3398&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=eic2QNLwx%2B18jVZHPfOGfA99%2B8gLaXXal2KuSBvi6s0%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 5 failures (97.500% success rate)
>>>
>>>
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3531&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=3apyNPTCcAbKQ55s1RqP7c%2BiuTjYWoDHN7itNiJBr6s%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3522&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=dsdzFOC4jPXN2pKibdZR8MWCwKEJWOCv8nNSnvrhclg%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3456&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=K2%2Fo2mdpIVC4u2V12VRkpyyGWJq2vYqtm1%2Fvki8mB9A%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=Dje%2F53pJYke0d5yosB61kMB77CLZYdLfqxWoD%2BfFzZs%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=1dKgH5KcHmj7snaRO46OW8k0GPrnTottsNCWH2HdQAE%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.ClientServerTransactionFailoverWithMixedVersionServersDistributedTest:
>>> 1 failures (99.500% success rate)
>>>
>>>
>>>
>>>
>>> clientTransactionOperationsAreNotLostIfTransactionIsOnRolledServer
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3433&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=fH3g67sitVbMU9Ir1Exm8MclDAVqM4x8mg%2F9y7WYi6I%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.wan.parallel.ParallelWANPersistenceEnabledGatewaySenderDUnitTest:
>>> 7 failures (96.500% success rate)
>>>
>>>
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3478&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=9JBfKJMKPrarh2IkhTzz%2BxxQ6Mdt0J2aaqHnn7fUsNo%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3463&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=Wi%2FM0H%2FEbUoS%2FEDaHs0ybqHqc5ZJY9jhai9XvGqOaeg%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3439&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=Dje%2F53pJYke0d5yosB61kMB77CLZYdLfqxWoD%2BfFzZs%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=ASOBOjhutihuLnyEk9v%2ByTlzeorAJrYFBlWKpIq3yVA%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3405&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=1dKgH5KcHmj7snaRO46OW8k0GPrnTottsNCWH2HdQAE%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3370&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=k1QhouuiRkGwHEoTve21cd1rClD6%2F3PEO%2Fp8pQUrX7A%3D&reserved=0
>>>
>>>
>>>
testpersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3351&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835822212&sdata=yDB750p04VE%2FuMwwjVpZ3zAtW50BlfdWUVFW205XnKw%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.partitioned.PersistentColocatedPartitionedRegionDistributedTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>> testMultipleColocatedChildPRsMissingWithSequencedStart
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3493&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=4jOpZq1Q6qXgNqHYQAWBW702yhCDpUHzQPmOKk1ef5A%3D&reserved=0
>>>
>>> testMissingColocatedChildPRDueToDelayedStart
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=P%2BZyelCzOwqc87bP9XFlIPdVYn0BwWk4jMIu0lxaq4M%3D&reserved=0
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3537&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=qt20QmQTmbe%2F33LtWmQsXnDWfIUB6vlf5xtE%2Bw23gG0%3D&reserved=0
>>>
>>> testHierarchyOfColocatedChildPRsMissingGrandchild
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3424&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=2mu0xJE6BaugPHzEheTw%2Fp4nDt6xdifAH0rCnF3mBno%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.distributed.DistributedMemberDUnitTest: 2 failures
>>> (99.000% success rate)
>>>
>>>
>>>
>>> testGroupsInAllVMs
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3423&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=VPQRY2DJxao%2F%2Fsj3rTMBtZ1J9R7FQYR7SqY1hgQoj1U%3D&reserved=0
>>>
>>> testGroupsInAllVMs
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3374&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=GyAnvRP4%2BVP4bYI%2BkXzrTgsVezpe0c6SKvnqiZq1p1k%3D&reserved=0
>>>
>>>
>>>
>>> org.apache.geode.management.JMXMBeanReconnectDUnitTest: 5 failures
>>> (97.500% success rate)
>>>
>>>
>>>
>>> serverMXBeansOnServerAreUnaffectedByLocatorCrash
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3503&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=P%2BZyelCzOwqc87bP9XFlIPdVYn0BwWk4jMIu0lxaq4M%3D&reserved=0
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3436&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=vq40%2BFWfDbH%2Fs4b%2BooGY3KBi1WV1LT%2Bh3s45BnAbq3I%3D&reserved=0
>>>
>>>
>>> serverMXBeansAreRestoredOnBothLocatorsAfterCrashedServerReturns
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3406&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=ZNwEvU4gWXhoVGVfx3vqHjHmV2%2FQHS1VOjifnY0qZCg%3D&reserved=0
>>>
>>> locatorHasMemberTypeMXBeansForBothLocators
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3352&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=gyq3c%2FGdfmE1OATwqe39MQUfjz%2Be6crV6dTDWQUgvUM%3D&reserved=0
>>>
>>> serverMXBeansOnLocatorAreRestoredAfterCrashedServerReturns
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3457&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=elpsPB4B3SCHvP0yrV0CuNMM11dAUr8yROnKa3RpQus%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.wan.offheap.SerialWANPersistenceEnabledGatewaySenderOffHeapDUnitTest:
>>> 4 failures (98.000% success rate)
>>>
>>>
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3494&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=tm2oxvbVR%2BcL7gPOqIOqIK0FKLbJr%2BEHrB4lywt649k%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3465&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=Ho8AWWGOORaKrtMlj38WaAs38BE5nmuaA1xGC%2B9UmCU%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3464&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835832204&sdata=QXxXqvCi7aDtvosglZ5q51fyAguonxny%2F5b%2FAFvjFL8%3D&reserved=0
>>>
>>>
>>>
testReplicatedRegionPersistentWanGateway_restartSenderWithCleanQueues_expectNoEventsReceived
>>>
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3379&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=7JCsVA7pAmjclJo2tk1jDq%2BCeMjHR2G2G6wLSfXCAqQ%3D&reserved=0
>>>
>>>
>>>
>>>
org.apache.geode.internal.cache.tier.sockets.ClientServerMiscSelectorDUnitTest:
>>> 3 failures (98.500% success rate)
>>>
>>>
>>>
>>> testPingWrongServer
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3402&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=d9GXmfDC%2BbpY21Pgat%2FIXRaJ61d3fWjhUCqvTwvNM9w%3D&reserved=0
>>>
>>> testPingWrongServer
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3380&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=WBz4zHqtAhx%2FBOy0lHi7vJWHgYzZHKNq7pynG7gt%2Frc%3D&reserved=0
>>>
>>> testPingWrongServer
>>>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconcourse.apachegeode-ci.info%2Fteams%2Fmain%2Fpipelines%2Fapache-mass-test-run-main%2Fjobs%2FDistributedTestOpenJDK8%2Fbuilds%2F3357&data=02%7C01%7Chansonm%40vmware.com%7C83cd55b2eaf54ce3cd6808d827820ea9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637302786835842199&sdata=99aY8oSD%2B2EF3RgdyZj3L9I2REpk7NPqbYuzz9SdIIA%3D&reserved=0
>>>
>>>
