[
https://issues.apache.org/jira/browse/HDFS-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841779#comment-17841779
]
ASF GitHub Bot commented on HDFS-17496:
---------------------------------------
hadoop-yetus commented on PR #6764:
URL: https://github.com/apache/hadoop/pull/6764#issuecomment-2081668506
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 89m 34s | | trunk passed |
| +1 :green_heart: | compile | 5m 58s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 04s | | trunk passed |
| +1 :green_heart: | shadedclient | 149m 03s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 39s | | the patch passed |
| +1 :green_heart: | compile | 3m 36s | | the patch passed |
| +1 :green_heart: | javac | 3m 36s | | the patch passed |
| -1 :x: | blanks | 0m 01s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6764/3/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 2m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 161m 52s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 24s | | The patch does not
generate ASF License warnings. |
| | | 427m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/6764 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 a2b679b71053 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 45aeb0964fdd0a6c8675fa2342d8893a600ed4fb |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6764/3/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6764/3/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> DataNode supports more fine-grained dataset lock based on blockid
> -----------------------------------------------------------------
>
> Key: HDFS-17496
> URL: https://issues.apache.org/jira/browse/HDFS-17496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Reporter: farmmamba
> Assignee: farmmamba
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-04-23-16-17-07-057.png
>
>
> Recently, we used NvmeSSD as volumes in datanodes and performed some stress
> tests.
> We found that NvmeSSD and HDD disks achieve similar performance when create
> lots of small files, such as 10KB.
> This phenomenon is counterintuitive. After analyzing the metric monitoring ,
> we found that fsdataset lock became the bottleneck in high concurrency
> scenario.
>
> Currently, we have two level locks which are BLOCK_POOL and VOLUME. We can
> further split the volume lock to DIR lock.
> DIR lock is defined as below： given a blockid, we can determine which subdir
> this block will be placed in finalized dir. We just use
> subdir[0-31]/subdir[0-31] as the
> name of DIR lock.
> More details, please refer to method DatanodeUtil#idToBlockDir：
> {code:java}
> public static File idToBlockDir(File root, long blockId) {
> int d1 = (int) ((blockId >> 16) & 0x1F);
> int d2 = (int) ((blockId >> 8) & 0x1F);
> String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
> DataStorage.BLOCK_SUBDIR_PREFIX + d2;
> return new File(root, path);
> } {code}
> The performance comparison is as below:
> experimental setup:
> 3 DataNodes with single disk.
> 10 Cients concurrent write and delete files after writing.
> 550 threads per Client.
> !image-2024-04-23-16-17-07-057.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
