[
https://issues.apache.org/jira/browse/GEODE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404105#comment-17404105
]
Dale Emery commented on GEODE-9531:
-----------------------------------
Looking at the test artifacts, I found:
* For some reason, too many tests were run concurrently. The job runs the
{{upgradeTest}} task with test task with {{-PdunitParallelForks=48}}, which
sets a limit of 48 concurrent tests, but Gradle somehow ran as many as 62 tests
concurrently. There were 60 running at the time of this failure.
* This failure happened on the 1.14 support branch. That branch did not include
my changes to upgrade Gradle and to run tests without Dockerizing them. So
those changes don't explain why too many tests ran concurrently.
> CI Failure: TxCommitMessageBCClientToServerTxPartitionTest fails with
> ForcedDisconnectException
> -----------------------------------------------------------------------------------------------
>
> Key: GEODE-9531
> URL: https://issues.apache.org/jira/browse/GEODE-9531
> Project: Geode
> Issue Type: Bug
> Affects Versions: 1.14.0
> Reporter: Donal Evans
> Assignee: Eric Shu
> Priority: Major
> Labels: GeodeOperationAPI
>
> {noformat}
> org.apache.geode.internal.cache.TxCommitMessageBCClientToServerTxPartitionTest
> > test[11] FAILED
> org.apache.geode.test.dunit.RMIException: While invoking
> org.apache.geode.internal.cache.TxCommitMessageBCTestBase$$Lambda$55/2050040059.run
> in VM 2 running on Host 1797ac7f43c4 with 5 VMs
> Caused by:
> org.apache.geode.distributed.DistributedSystemDisconnectedException:
> membership shutdown, caused by org.apache.geode.ForcedDisconnectException:
> Member isn't responding to heartbeat requests
> Caused by:
> org.apache.geode.ForcedDisconnectException: Member isn't
> responding to heartbeat requests
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> -----------------------------------------------------------------------
> Found suspect string in 'dunit_suspect-vm2.log' at line 993
> [fatal 2021/05/25 16:58:13.700 GMT <unicast receiver,1797ac7f43c4-36391>
> tid=1349] Membership service failure: Member isn't responding to heartbeat
> requests
>
> org.apache.geode.distributed.internal.membership.api.MemberDisconnectedException:
> Member isn't responding to heartbeat requests
> at
> org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.forceDisconnect(GMSMembership.java:1783)
> at
> org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:1122)
> at
> org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveMemberMessage(GMSJoinLeave.java:725)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1366)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1302)
> at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
> at org.jgroups.JChannel.up(JChannel.java:741)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:72)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:70)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1876)
> at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1789)
> at org.jgroups.protocols.TP.receive(TP.java:1714)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:159)
> at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
> at java.lang.Thread.run(Thread.java:748)
> -----------------------------------------------------------------------
> Found suspect string in 'dunit_suspect-vm2.log' at line 1041
> [error 2021/05/25 16:58:14.206 GMT <RMI TCP Connection(23)-172.17.0.39>
> tid=135] Cache initialization for GemFireCache[id = 664332017; isClosing =
> false; isShutDownAll = false; created = Tue May 25 16:57:54 GMT 2021; server
> = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] failed
> because:
> org.apache.geode.distributed.DistributedSystemDisconnectedException:
> membership shutdown, caused by org.apache.geode.ForcedDisconnectException:
> Member isn't responding to heartbeat requests
> at
> org.apache.geode.distributed.internal.DistributionImpl.checkCancelled(DistributionImpl.java:313)
> at
> org.apache.geode.distributed.internal.DistributionImpl.send(DistributionImpl.java:243)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2053)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:1981)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2018)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1083)
> at
> org.apache.geode.internal.cache.CreateRegionProcessor.initializeRegion(CreateRegionProcessor.java:115)
> at
> org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1161)
> at
> org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1092)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
> at
> org.apache.geode.internal.cache.InternalRegionFactory.create(InternalRegionFactory.java:78)
> at
> org.apache.geode.pdx.internal.PeerTypeRegistration.initialize(PeerTypeRegistration.java:202)
> at
> org.apache.geode.pdx.internal.TypeRegistry.initialize(TypeRegistry.java:116)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initializePdxRegistry(GemFireCacheImpl.java:1671)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1605)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1448)
> at
> org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
> at
> org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
> at
> org.apache.geode.internal.cache.TxCommitMessageBCTestBase.createServerCacheWithPool(TxCommitMessageBCTestBase.java:187)
> at
> org.apache.geode.internal.cache.TxCommitMessageBCTestBase.lambda$postSetUp$384cd611$1(TxCommitMessageBCTestBase.java:117)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
> at
> org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
> at sun.reflect.GeneratedMethodAccessor333.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.geode.ForcedDisconnectException: Member isn't
> responding to heartbeat requests
> at
> org.apache.geode.distributed.internal.DistributionImpl.checkCancelled(DistributionImpl.java:312)
> ... 42 more
> -----------------------------------------------------------------------
> Found suspect string in 'dunit_suspect-vm2.log' at line 1130
> [error 2021/05/25 16:58:15.274 GMT <RMI TCP Connection(23)-172.17.0.39>
> tid=135] org.apache.geode.distributed.DistributedSystemDisconnectedException:
> membership shutdown, caused by org.apache.geode.ForcedDisconnectException:
> Member isn't responding to heartbeat requests
> 576 tests completed, 1 failed, 36 skipped
> {noformat}
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-= Test Results URI
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> http://files.apachegeode-ci.info/builds/apache-support-1-14-main/1.14.0-build.0787/test-results/upgradeTest/1621966586/
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Test report artifacts from this job are available at:
> http://files.apachegeode-ci.info/builds/apache-support-1-14-main/1.14.0-build.0787/test-artifacts/1621966586/upgradetestfiles-OpenJDK8-1.14.0-build.0787.tgz
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
