[
https://issues.apache.org/jira/browse/HDFS-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836178#comment-17836178
]
ASF GitHub Bot commented on HDFS-17458:
---------------------------------------
hfutatzhanghb commented on PR #6717:
URL: https://github.com/apache/hadoop/pull/6717#issuecomment-2049700613
> @hfutatzhanghb Thanks for your works. We should be careful to remove BP
lock here. List one of the changes as example, it will return one definite
value before this PR because hold RW lock here, but uncertain after this PR,
such as another thread invoke `map.put` between `map.get` and `return` it will
return null, but if invoke `map.put` before them it will return one
`ReplicaInfo` object.
>
> ```
> ReplicaInfo get(String bpid, long blockId) {
> checkBlockPool(bpid);
> - try (AutoCloseDataSetLock l =
lockManager.readLock(LockLevel.BLOCK_POOl, bpid)) {
> - LightWeightResizableGSet<Block, ReplicaInfo> m = map.get(bpid);
> - return m != null ? m.get(new Block(blockId)) : null;
> - }
> + LightWeightResizableGSet<Block, ReplicaInfo> m = map.get(bpid);
> + return m != null ? m.get(new Block(blockId)) : null;
> }
> ```
>
> I didn't traverse all invoker here, and not sure if it will involve some
potential risk. FYI.
Sir, Thanks for your replying. Yes, we need to be very careful to modify
class ReplicaMap. In fact, i have check the methods one by one and I think we
can push this PR forward after it runs stablely on our product for a long time.
> Remove unnecessary BP lock in ReplicaMap
> ----------------------------------------
>
> Key: HDFS-17458
> URL: https://issues.apache.org/jira/browse/HDFS-17458
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.4.0
> Reporter: farmmamba
> Assignee: farmmamba
> Priority: Major
> Labels: pull-request-available
>
> In HDFS-16429 we make LightWeightResizableGSet to be thread safe, and in
> HDFS-16511 we change some methods in ReplicaMap to acquire read lock instead
> of acquiring write lock.
> This PR try to remove unnecessary Block_Pool read lock further.
> Recently, I performed stress tests on datanodes to measure their read/write
> operations/second.
> Before we removing some lock, it can only achieve ~2K write ops. After
> optimizing, it can achieve more than 5K write ops.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
