[
https://issues.apache.org/jira/browse/HDFS-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911302#comment-17911302
]
ASF GitHub Bot commented on HDFS-17496:
---------------------------------------
hadoop-yetus commented on PR #7279:
URL: https://github.com/apache/hadoop/pull/7279#issuecomment-2579053966
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 19s | |
https://github.com/apache/hadoop/pull/7279 does not apply to trunk. Rebase
required? Wrong Branch? See
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
|
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/7279 |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7279/1/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> DataNode supports more fine-grained dataset lock based on blockid
> -----------------------------------------------------------------
>
> Key: HDFS-17496
> URL: https://issues.apache.org/jira/browse/HDFS-17496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Reporter: farmmamba
> Assignee: farmmamba
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
> Attachments: image-2024-04-23-16-17-07-057.png
>
>
> Recently, we used NvmeSSD as volumes in datanodes and performed some stress
> tests.
> We found that NvmeSSD and HDD disks achieve similar performance when create
> lots of small files, such as 10KB.
> This phenomenon is counterintuitive. After analyzing the metric monitoring ,
> we found that fsdataset lock became the bottleneck in high concurrency
> scenario.
>
> Currently, we have two level locks which are BLOCK_POOL and VOLUME. We can
> further split the volume lock to DIR lock.
> DIR lock is defined as below： given a blockid, we can determine which subdir
> this block will be placed in finalized dir. We just use
> subdir[0-31]/subdir[0-31] as the
> name of DIR lock.
> More details, please refer to method DatanodeUtil#idToBlockDir：
> {code:java}
> public static File idToBlockDir(File root, long blockId) {
> int d1 = (int) ((blockId >> 16) & 0x1F);
> int d2 = (int) ((blockId >> 8) & 0x1F);
> String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
> DataStorage.BLOCK_SUBDIR_PREFIX + d2;
> return new File(root, path);
> } {code}
> The performance comparison is as below:
> experimental setup:
> 3 DataNodes with single disk.
> 10 Cients concurrent write and delete files after writing.
> 550 threads per Client.
> !image-2024-04-23-16-17-07-057.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
