[
https://issues.apache.org/jira/browse/HDFS-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18064462#comment-18064462
]
ASF GitHub Bot commented on HDFS-17890:
---------------------------------------
junjie1233 opened a new pull request, #8338:
URL: https://github.com/apache/hadoop/pull/8338
## Summary
When a client requests to read a block, the NameNode returns a list of
DataNodes holding the replicas of that block.
The current logic sorts these DataNodes based on network topology (rack
awareness, distance), without considering the performance of the underlying
storage (disk/volume).
If a block replica resides on a slow or overloaded disk (for example, a hot
disk with high latency), its DataNode may still be placed at the top of the
sorted list and selected first by the client.
---
## Example
Suppose a block has three replicas located on:
dn1: storage1 (slow disk)
dn2: storage1 (normal disk)
dn3: storage1 (normal disk)
Even if the speed of memory 1 is known to be slow, the client will still
prioritize reading in the order returned by the NameNode.
---
## Fix
This PR adds **disk-level slow storage tracking** and **deprioritization**
during block location sorting.
### 1. Disk-Level Tracking (`SlowDiskTracker.java`)
- Track slow storage at **StorageID granularity** (not DataNode level).
- Copy-on-Write cache `cachedSlowDisksForRead` to avoid lock contention on
read path.
- **Cache key**: `IP:PORT:StorageID` (e.g., `127.0.0.1:50010:DS-xxx`).
- Dual key modes:
- **CACHE_KEY**: `IP:PORT:StorageID` for read deprioritization.
- **LEGACY_KEY**: `IP:PORT:volumeName` for backward compatibility with
Top-N reports.
- Background cache rebuild with configurable interval (default **30s**).
- Automatic expiration of stale entries (`reportValidityMs`).
### 2. Block Location Sorting (`FSNamesystem.java`)
- New method `sortLocatedBlocksBySlowDisk()` reorders replicas **after
topology-based sorting**.
- Pre-compute slow keys before sorting to avoid string concatenation in
comparator hot path.
- **Stable sort**: preserves network topology order for non-slow replicas.
- Slow replicas sorted by latency (**higher latency → lower priority**).
- Controlled by config:
`dfs.namenode.deprioritize.slow.disk.datanode.for.read` (default:
**false**).
### 3. Configuration (`DFSConfigKeys.java`)
- `dfs.namenode.slow.disk.cache.rebuild.interval` (default **30s**).
- Decouples cache rebuild frequency from Top-N report generation interval.
- Allows independent tuning for large clusters.
### 4. Disk Key Format (`DataNodeDiskMetrics.java`)
- Use `volumeName|storageID` format for slow disk reports.
- Enables `SlowDiskTracker` to extract both **legacy key** (WebUI) and
**cache key** (read path).
---
## Test
**Test class**: `TestSlowDiskBlockLocations.java`
### Test Coverage
- ✅ **testDeprioritizeSlowDiskDatanodeForReadEnabled**
Verifies that slow disk replicas are moved to the end of location list.
Checks block read path integration.
- ✅ **testSlowDiskCacheRebuild**
Tests cache population after DataNode reports slow disk.
Verifies cache refresh mechanism.
- ✅ **testSlowDiskExpiration**
Validates expiration of stale slow disk entries.
Confirms cache is cleaned after disk recovery.
- ✅ **testCacheIntegrationWithReadPath**
End-to-end test: slow disk report → cache update → block location sorting.
Verifies clients avoid slow replicas.
- ✅ **testIndependentCacheRebuildInterval**
Tests independent cache rebuild interval configuration.
Verifies decoupling from Top-N report generation.
- ✅ **testMultipleSlowDisks**
Multiple slow disks across different DataNodes.
Validates sorting by latency when all replicas are slow.
- ✅ **testNoSlowDiskReports**
Baseline test: no sorting when no slow disks reported.
Ensures feature is non-intrusive when disabled.
---
### Test Configuration
- `DFS_HEARTBEAT_INTERVAL`: **1s** (fast heartbeat for testing)
- `DFS_NAMENODE_SLOW_DISK_CACHE_REBUILD_INTERVAL`: **1s** (quick cache
rebuild)
- `OUTLIERS_REPORT_INTERVAL`: **1s** (rapid slow disk detection)
- Uses `GenericTestUtils.waitFor()` for async operations.
> Avoid slow disks datanode when reading data
> -------------------------------------------
>
> Key: HDFS-17890
> URL: https://issues.apache.org/jira/browse/HDFS-17890
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Reporter: JunJieZhao
> Priority: Major
>
> During HDFS read operations, replicas on slow disks should be deprioritized
> so that replicas on healthy disks are preferred.
> Currently, slow disk avoidance is not performed at the disk level when
> reading operations return to the block position.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
