[
https://issues.apache.org/jira/browse/GEODE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Shu resolved GEODE-5888.
-----------------------------
Resolution: Fixed
Fix Version/s: 1.8.0
> A removeAll/putAll operation could remove lockObject held by another thread
> if region is closing
> ------------------------------------------------------------------------------------------------
>
> Key: GEODE-5888
> URL: https://issues.apache.org/jira/browse/GEODE-5888
> Project: Geode
> Issue Type: Bug
> Components: regions
> Affects Versions: 1.1.0
> Reporter: Eric Shu
> Assignee: Eric Shu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A bulk operation needs to hold lockObjects in BucketRegion.waitUntilLocked.
> However, it may hit RegionDestroyedException instead, and it will release
> lockObjects it does not hold.
> org.apache.geode.cache.RegionDestroyedException: Partitioned Region @21554698
> [path='/partitionedRegion'; dataPolicy=PERSISTENT_PARTITION; prId=1;
> isDestroyed=true; isClosed=true; retryTimeout=3600000; serialNumber=7;
> partition
> attributes=PartitionAttributes@2144398013[redundantCopies=0;localMaxMemory=205;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=null;recoveryDelay=-1;startupRecoveryDelay=0;FixedPartitionAttributes=null;partitionListeners=null];
> on VM
> rs-GEM-1352-1526a0i3large-hydra-client-34(dataStoregemfire4_host1_19090:19090)<ec><v23>:1028]
> at
> org.apache.geode.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:7396)
> at
> org.apache.geode.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2745)
> at
> org.apache.geode.internal.cache.BucketRegion.waitUntilLocked(BucketRegion.java:493)
> at
> org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.doLocalRemoveAll(RemoveAllPRMessage.java:437)
> at
> org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.operateOnPartitionedRegion(RemoveAllPRMessage.java:317)
> at
> org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:325)
> at
> org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:367)
> at
> org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:424)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2873)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:2553)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$1300(ClusterDistributionManager.java:109)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.messageReceived(ClusterDistributionManager.java:3411)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1102)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1022)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:399)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:722)
> at
> org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:824)
> at
> org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3898)
> at
> org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3508)
> at
> org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1814)
> at org.apache.geode.internal.tcp.Connection.run(Connection.java:1676)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Once the exception is hit, it will remove the lockObjects hold by another
> thread performing another operation.
> at
> org.apache.geode.internal.cache.BucketRegion.removeAndNotifyKeys(BucketRegion.java:454)
> at
> org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.doLocalRemoveAll(RemoveAllPRMessage.java:534)
> at
> org.apache.geode.internal.cache.partitioned.RemoveAllPRMessage.operateOnPartitionedRegion(RemoveAllPRMessage.java:317)
> at
> org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:325)
> at
> org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:367)
> at
> org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:424)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2873)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:2553)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$1300(ClusterDistributionManager.java:109)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.messageReceived(ClusterDistributionManager.java:3411)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1102)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1022)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:399)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:722)
> at
> org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:824)
> at
> org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3898)
> at
> org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3508)
> at
> org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1814)
> at org.apache.geode.internal.tcp.Connection.run(Connection.java:1676)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> This causes a race condition that a new operation could come in and work on a
> region entry that the other thread is working on but the lockObject of the
> region entry was removed by the thread hitting the RegionDestroyedException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
