[
https://issues.apache.org/jira/browse/GEODE-8206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17121124#comment-17121124
]
Bruce J Schuchardt commented on GEODE-8206:
-------------------------------------------
The test hung trying to shut down the locators it started. The teardown thread
was stuck waiting on cache initialization:
{noformat}
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0e6cb10> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
at
org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4693)
at
org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4671)
at
org.apache.geode.internal.cache.LocalRegion.getSubregion(LocalRegion.java:4558)
at
org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3322)
at
org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3153)
at
org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.getConfigurationRegion(InternalConfigurationPersistenceService.java:792)
at
org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.destroySharedConfiguration(InternalConfigurationPersistenceService.java:639)
at
org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest.lambda$teardown$bb17a952$1(ReconnectWithClusterConfigurationDUnitTest.java:112)
at
org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest$$Lambda$339/1176989958.run(Unknown
Source)
{noformat}
One of the locators was in a loop trying to reconnect and failing over and over
again. I think this was due to misconfiguration of the locators. There are
two locators and they had this for their "locators" configuration:
{noformat}
locators=localHost[27228],localHost[27228]
{noformat}
This was due to faulty code in the test's getDistributedSystemProperties method
that only told the locators about the first locator's port.
{code}
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("localHost[")
.append(locatorPorts[0])
.append(']');
for (int i = 1; i < NUM_LOCATORS; i++) {
stringBuilder.append(",localHost[")
.append(locatorPorts[0])
.append(']');
}
{code}
The loop should use "i", not "0" for an index into localPorts[].
> CI Failure:
> ReconnectWithClusterConfigurationDUnitTest.testReconnectAfterMeltdown hang
> --------------------------------------------------------------------------------------
>
> Key: GEODE-8206
> URL: https://issues.apache.org/jira/browse/GEODE-8206
> Project: Geode
> Issue Type: Bug
> Components: ci, membership
> Reporter: Eric Shu
> Priority: Major
>
> This test hangs in:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/221#A
> {noformat}
> "RMI TCP Connection(1)-172.17.0.23" #32 daemon prio=5 os_prio=0
> tid=0x00007fea58001800 nid=0x27d waiting on condition [0x00007feb23bf7000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000e0e6cb10> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
> at
> org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4693)
> at
> org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4671)
> at
> org.apache.geode.internal.cache.LocalRegion.getSubregion(LocalRegion.java:4558)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3322)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3153)
> at
> org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.getConfigurationRegion(InternalConfigurationPersistenceService.java:792)
> at
> org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.destroySharedConfiguration(InternalConfigurationPersistenceService.java:639)
> at
> org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest.lambda$teardown$bb17a952$1(ReconnectWithClusterConfigurationDUnitTest.java:112)
> at
> org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest$$Lambda$339/1176989958.run(Unknown
> Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
> at
> org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$15/2049270388.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000e0a75928> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> It is waiting for initializationLatchAfterGetInitialImage to countdown. Not
> sure why gii was not able to finish.
> There is another reconnect thread is waiting in another process. I do not
> have enough context to tell if it is the gii provider or not.
> {noformat}
> "ReconnectThread" #77 prio=5 os_prio=0 tid=0x00007f42281f7000 nid=0x353 in
> Object.wait() [0x00007f43082db000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2568)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2423)
> - locked <0x00000000e0867ca8> (a java.lang.Object)
> - locked <0x00000000e08b1778> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> - locked <0x00000000e0867ce8> (a java.lang.Class for
> org.apache.geode.internal.cache.InternalCacheBuilder)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1275)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:2315)
> at
> org.apache.geode.distributed.internal.membership.gms.GMSMembership.uncleanShutdown(GMSMembership.java:1287)
> at
> org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.lambda$forceDisconnect$0(GMSMembership.java:2030)
> at
> org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl$$Lambda$261/2070151959.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
