[
https://issues.apache.org/jira/browse/HDFS-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18058328#comment-18058328
]
ASF GitHub Bot commented on HDFS-17372:
---------------------------------------
hfutatzhanghb opened a new pull request, #6530:
URL: https://github.com/apache/hadoop/pull/6530
### Description of PR
Refer to HDFS-17372.
Recently, we met a critical problem in our production cluster which have
lots of small files. In that cluster, per datanode has almost six million
blocks.
After deleting large dir or recommision, some datanodes's
SumOfActorCommandQueueLength metrics became very high as below picture shows.
After a while, we found some datanodes's write block ops became zero. That
means client can not write to those datanodes. We found some logs on those
datanodes:
```
2024-02-05 17:30:11,657 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Block token verification
failed: op=WRITE_BLOCK, remoteAddress=/x.x.x.x:63990, message=Can't re-compute
password for block_token_identifier (expiryDate=1707161411033, keyId=596193458,
userId=hdfs, blockPoolId=BP-1169917699-x.x.x.x-1678688680604,
blockId=11354988786, access modes=[WRITE], storageTypes= [DISK, DISK, DISK,
DISK, DISK, DISK, DISK], storageIds= [DS-2664b73d-1cc9-4613-b6e3-cd58ec5ae8d5,
DS-8f18621e-46c1-49b3-8bf5-7f5b8e47fe80,
DS-527817e1-c5d0-4085-929f-9100a423223f,
DS-d3f850c9-18ab-43cc-86d9-779b9d71a79f,
DS-e4aa15ce-0c5f-4645-a582-e187e46e7183,
DS-3fbbd401-58d2-49ac-a258-da7769417593,
DS-d76a51e4-73db-4db6-a4e0-f82a38c1ad82]), since the required block key
(keyID=596193458) doesn't exist.
```
That is to say, DNA_ACCESSKEYUPDATE command was blocked in
CommandProcessingThread#queue. This can be deadly.
So, we should guarantee that command with high-priority should be processed
in time.
> CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high
> priority command blocked by low priority command
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17372
> URL: https://issues.apache.org/jira/browse/HDFS-17372
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Haobo Zhang
> Assignee: Haobo Zhang
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
