[ https://issues.apache.org/jira/browse/GEODE-8825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359662#comment-17359662 ]
Barrett Oglesby commented on GEODE-8825: ---------------------------------------- Here is some logging that shows the behavior: Creating the receiver causes it to get added to the added to federatedComponentMap: {noformat} [vm1] [warn 2021/06/08 16:36:38.288 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy about to create receiver [vm1] [warn 2021/06/08 16:36:38.370 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX LocalManager.markForFederation added to federatedComponentMap objName=GemFire:service=GatewayReceiver [vm1] [warn 2021/06/08 16:36:38.376 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy created receiver {noformat} There is no logging here, but the Management Task hasn't run when the receiver is destroyed. The mbean is removed from federatedComponentMap, but since the monitoringRegion doesn't contain the mbean, it doesn't get removed from the region. The Management Task adds the mbean to that region (which is how it gets to the manager). {noformat} [vm1] [warn 2021/06/08 16:36:38.382 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy about to destroy receiver [vm1] [warn 2021/06/08 16:36:38.384 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX LocalManager.unMarkForFederation removed from federatedComponentMap objName=GemFire:service=GatewayReceiver [vm1] [warn 2021/06/08 16:36:38.388 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX LocalManager.unMarkForFederation monitoringRegionContains objName=GemFire:service=GatewayReceiver: false [vm1] [warn 2021/06/08 16:36:38.389 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy destroyed receiver {noformat} If I add a sleep between the create and destroy, I see better behavior. Here is some logging that shows that. The receiver is created the same as before: {noformat} [vm1] [warn 2021/06/08 16:35:40.970 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy about to create receiver [vm1] [warn 2021/06/08 16:35:41.054 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX LocalManager.markForFederation added to federatedComponentMap objName=GemFire:service=GatewayReceiver,type=Member,member=192.168.1.4(12942)<v2>-41002 [vm1] [warn 2021/06/08 16:35:41.061 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy created receiver {noformat} The Management Task puts the map into the monitoringRegion adds the proxy: {noformat} [vm1] [warn 2021/06/08 16:35:41.775 PDT <Management Task1> tid=0x3e] XXX LocalManager.doManagementTask about to put replicaMap={GemFire:service=GatewayReceiver = GemFire:service=GatewayReceiver} [vm0] [warn 2021/06/08 16:35:41.782 PDT <P2P message reader for 192.168.1.4(12942)<v2>:41002 unshared ordered sender uid=6 dom #1 local port=60249 remote port=54707> tid=0x48] XXX MBeanAggregator.afterCreateProxy objectName=GemFire:service=GatewayReceiver,type=Member,member=192.168.1.4(12942)<v2>-41002 {noformat} The receiver is destroyed. This time, the monitoringRegion contains the mbean, so it is removed from it: {noformat} [vm1] [warn 2021/06/08 16:35:44.072 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy about to destroy receiver [vm1] [warn 2021/06/08 16:35:44.074 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX ManagementAdapter.handleGatewayReceiverDestroy objectName=GemFire:service=GatewayReceiver [vm1] [warn 2021/06/08 16:35:44.075 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX LocalManager.unMarkForFederation removed from federatedComponentMap objName=GemFire:service=GatewayReceiver [vm1] [warn 2021/06/08 16:35:44.075 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX LocalManager.unMarkForFederation monitoringRegionContains objName=GemFire:service=GatewayReceiver: true [vm1] [warn 2021/06/08 16:35:44.079 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX LocalManager.unMarkForFederation removed from monitoringRegion objName=GemFire:service=GatewayReceiver [vm1] [warn 2021/06/08 16:35:44.079 PDT <RMI TCP Connection(1)-127.0.0.1> tid=0x13] XXX GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy destroyed receiver {noformat} The proxy is removed from the mananger: {noformat} [vm0] [warn 2021/06/08 16:35:44.082 PDT <P2P message reader for 192.168.1.4(12942)<v2>:41002 unshared ordered sender uid=4 dom #1 local port=60249 remote port=54691> tid=0x41] XXX MBeanAggregator.afterRemoveProxy objectName=GemFire:service=GatewayReceiver,type=Member,member=192.168.1.4(12942)<v2>-41002 {noformat} The test needs to be modified to wait for the manager to contain the proxy before destroying the receiver. > CI failure: GatewayReceiverMBeanDUnitTest > > testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy > ---------------------------------------------------------------------------------------------------- > > Key: GEODE-8825 > URL: https://issues.apache.org/jira/browse/GEODE-8825 > Project: Geode > Issue Type: Bug > Components: tests, wan > Reporter: Jianxia Chen > Priority: Major > Labels: flaky > > {code:java} > org.apache.geode.internal.cache.wan.GatewayReceiverMBeanDUnitTest > > testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy FAILED > org.apache.geode.test.dunit.RMIException: While invoking > org.apache.geode.internal.cache.wan.GatewayReceiverMBeanDUnitTest$$Lambda$202/0x00000001008f0c40.run > in VM 0 running on Host c3e48bdac460 with 4 VMs > at org.apache.geode.test.dunit.VM.executeMethodOnObject(VM.java:623) > at org.apache.geode.test.dunit.VM.invoke(VM.java:447) > at > org.apache.geode.internal.cache.wan.GatewayReceiverMBeanDUnitTest.testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy(GatewayReceiverMBeanDUnitTest.java:76) > Caused by: > java.lang.AssertionError: expected null, but was:<ObjectName = > GemFire:service=GatewayReceiver,type=Member,member=172.17.0.18(183)<v10>-41002> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotNull(Assert.java:756) > at org.junit.Assert.assertNull(Assert.java:738) > at org.junit.Assert.assertNull(Assert.java:748) > at > org.apache.geode.internal.cache.wan.GatewayReceiverMBeanDUnitTest.verifyMBeanProxiesDoesNotExist(GatewayReceiverMBeanDUnitTest.java:106) > at > org.apache.geode.internal.cache.wan.GatewayReceiverMBeanDUnitTest.lambda$testMBeanAndProxiesForGatewayReceiverAreRemovedOnDestroy$bb17a952$3(GatewayReceiverMBeanDUnitTest.java:76) > {code} > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK11/builds/704 > =-=-=-=-=-=-=-=-=-=-=-=-=-=-= Test Results URI > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > http://files.apachegeode-ci.info/builds/apache-develop-main/1.14.0-build.0601/test-results/distributedTest/1610390301/ > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Test report artifacts from this job are available at: > http://files.apachegeode-ci.info/builds/apache-develop-main/1.14.0-build.0601/test-artifacts/1610390301/distributedtestfiles-OpenJDK11-1.14.0-build.0601.tgz -- This message was sent by Atlassian Jira (v8.3.4#803005)