The main thread has locked the CacheFactory (0x000000021de38d70) and is initializing the cache. At this time it is registering a partitioned region by getting a distributed lock. It is waiting for a reply to this request from a remote member:
"main" #1 prio=5 os_prio=0 tid=0x00007f855c00c800 nid=0x1763e waiting on condition [0x00007f8563ca2000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000001c9d4b2e8> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64) at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715) at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:795) at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771) at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:858) at org.apache.geode.distributed.internal.locks.DLockRequestProcessor.requestLock(DLockRequestProcessor.java:242) at org.apache.geode.distributed.internal.locks.DLockService.lockInterruptibly(DLockService.java:1537) at org.apache.geode.distributed.internal.locks.DLockService.lock(DLockService.java:1260) at org.apache.geode.internal.cache.PartitionedRegion$BucketLock.basicLock(PartitionedRegion.java:6683) at org.apache.geode.internal.cache.PartitionedRegion$BucketLock.lock(PartitionedRegion.java:6626) at org.apache.geode.internal.cache.PartitionedRegion$RegionLock.lock(PartitionedRegion.java:6883) at org.apache.geode.internal.cache.PartitionedRegion.registerPartitionedRegion(PartitionedRegion.java:1318) at org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:987) - locked <0x000000021dd35258> (a org.apache.geode.internal.cache.PartitionedRegion) at org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1157) at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104) at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:3004) at org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:243) at org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:558) at org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:512) at org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:338) at org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4291) at org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1390) at org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1195) at org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:758) - locked <0x000000021de39cf0> (a java.lang.Class for org.apache.geode.internal.cache.GemFireCacheImpl) at org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:745) at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:173) - locked <0x000000021de38d70> (a java.lang.Class for org.apache.geode.cache.CacheFactory) at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:212) - locked <0x000000021de38d70> (a java.lang.Class for org.apache.geode.cache.CacheFactory) at org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52) at org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:860) at org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:772) at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:699) at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:231) In the meantime, a PutMessage is being processed by a P2P message reader thread. This is a send of a primary put from another server. As part of that put operation, its delivering the event to a gateway sender which ultimately causes a VSDCountersManager to be instantiated. This is blocked waiting for the CacheFactory. The CacheFactory won't be available until the cache initialization is complete. "P2P message reader for 10.218.110.60(sbimgapp15-server2:1746)<v7>:1024 shared ordered uid=140 port=55882" #403 daemon prio=10 os_prio=0 tid=0x00007f84e8010800 nid=0x1a992 waiting for monitor entry [0x00007f84d0f9a000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.geode.cache.CacheFactory.getAnyInstance(CacheFactory.java:282) - waiting to lock <0x000000021de38d70> (a java.lang.Class for org.apache.geode.cache.CacheFactory) at amdocs.imdg.statistics.GemFireStatisticsFactory.getStatisticsFactory(GemFireStatisticsFactory.java:43) at amdocs.imdg.statistics.VSDCountersManager.<init>(VSDCountersManager.java:35) at amdocs.imdg.statistics.VSDCountersManager.<clinit>(VSDCountersManager.java:19) at amdocs.imdg.statistics.CountersManagerFactory.getCountersManager(CountersManagerFactory.java:27) at amdocs.imdg.utils.pooling.DataPoolFactory.makeObject(DataPoolFactory.java:42) at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797) - locked <0x000000021ed39f00> (a org.apache.commons.pool.impl.GenericKeyedObjectPool) at amdocs.imdg.utils.pooling.DataPool$DataPoolManager.getByteArray(DataPool.java:236) at amdocs.imdg.utils.pooling.DataPool.getByteArray(DataPool.java:98) at amdocs.imdg.model.BusinessData.populateData(BusinessData.java:110) at amdocs.imdg.utils.FlatBuffersUtils.updateBusinessData(FlatBuffersUtils.java:2255) at amdocs.imdg.utils.FlatBuffersUtils.updateCustomerData(FlatBuffersUtils.java:3083) at amdocs.imdg.utils.FlatBuffersUtils.updateNewCustomer(FlatBuffersUtils.java:3103) at amdocs.imdg.utils.FlatBuffersUtils.updateFromCustomerData(FlatBuffersUtils.java:2798) at amdocs.imdg.model.Customer.fromData(Customer.java:696) - locked <0x00000002469dd138> (a amdocs.imdg.model.Customer) at org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2372) at org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2395) at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813) at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961) at org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99) at org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911) at org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904) at org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134) - locked <0x000000021e7b50c8> (a org.apache.geode.internal.cache.VMCachedDeserializable) at org.apache.geode.internal.cache.EntryEventImpl.lambda$getNewValue$1(EntryEventImpl.java:975) at org.apache.geode.internal.cache.EntryEventImpl$$Lambda$41/406866714.apply(Unknown Source) at org.apache.geode.internal.cache.EntryEventImpl.callWithOffHeapLock(EntryEventImpl.java:1007) at org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:970) at amdocs.imdg.listeners.sbg.SGInfoSubstitutionFilter.getSubstituteValue(SGInfoSubstitutionFilter.java:37) at org.apache.geode.internal.cache.wan.AbstractGatewaySender.getSubstituteValue(AbstractGatewaySender.java:1093) at org.apache.geode.internal.cache.wan.AbstractGatewaySender.distribute(AbstractGatewaySender.java:971) at org.apache.geode.internal.cache.LocalRegion.notifyGatewaySender(LocalRegion.java:6234) at org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5928) at org.apache.geode.internal.cache.partitioned.PutMessage.operateOnPartitionedRegion(PutMessage.java:786) at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332) at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374) at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:432) at org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3571) at org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3205) at org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4384) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1117) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1041) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:408) at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714) at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:874) at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3966) at org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3552) at org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1828) at org.apache.geode.internal.tcp.Connection.run(Connection.java:1689) at java.lang.Thread.run(Thread.java:745) Then, there are 9 more P2P message reader threads waiting for the GenericKeyedObjectPool (0x000000021ed39f00) held by this thread. I'm not sure whether or not this is a deadlock. It would be interesting to see additional thread dumps for this server to see if the main thread is processing or stuck and also thread dumps for the other server(s) to see specifically what the thread processing the DLockRequestMessage is doing (thats the remote thread that the main thread above is waiting for). Would it be possible to create the VSDCountersManager in the main thread before the P2P message reader thread attempts to do it (like in a Function init)? Thanks, Barry Oglesby On Wed, Oct 24, 2018 at 7:40 AM Anthony Baker <aba...@pivotal.io> wrote: > Sorry I don’t think I understand your question. Did you encounter a > problem with getAnyInstance()? > > Anthony > > > > On Oct 24, 2018, at 1:27 AM, Dinesh Akhand <dines...@amdocs.com> wrote: > > > > No dead local was there , reader thread are waiting mostly. > > PFA whole stack strace. > > > > Thanks, > > Dinesh Akhand > >
