[
https://issues.apache.org/jira/browse/HDFS-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111767#comment-18111767
]
ASF GitHub Bot commented on HDFS-17977:
---------------------------------------
rdhabalia opened a new pull request, #8719:
URL: https://github.com/apache/hadoop/pull/8719
### Description of PR
`TestDFSIO` is the standard HDFS I/O benchmark, but it is a poor fit for
targeted DataNode stress testing:
- It launches a MapReduce job scheduled by YARN across the whole cluster, so
it cannot generate controlled, steady QPS/throughput and takes a long time
to saturate a targeted subset of nodes.
- It cannot target a specific set of DataNodes, such as a single replica set,
which is required to reproduce and measure hot or overloaded nodes.
- It reports aggregate throughput only and does not provide client-side
latency distributions such as p50/p95/p99, which are essential for
characterizing tail latency and disk random-I/O behavior.
- It requires the MapReduce/YARN stack to be available.
## Approach
Add `HdfsStressTest`, a single-process, no-MapReduce load generator.
Because it depends only on the HDFS client (`DistributedFileSystem` plus
favored-nodes) and does not require MapReduce/YARN, it lives in the
`hadoop-hdfs` test tree (`org.apache.hadoop.hdfs`), alongside other
standalone
HDFS benchmarks such as `BenchmarkThroughput`, rather than in the MapReduce
jobclient module.
The tool is intentionally compact, consisting of one tool class with small
nested helpers, and provides the controls that `TestDFSIO` lacks:
- **Controlled read/write throughput**
- Uses a global token-bucket rate limiter.
- Supports an optional linear ramp from start to end throughput for
acceleration stress testing.
- **Targeted DataNodes**
- Uses HDFS favored-nodes hints so that load is directed to a chosen
replica set.
- **Configurable block and file size**
- Supports configurable block size.
- Supports configurable test file size.
- **Cold-read testing**
- A pre-test phase creates a large corpus of files sized to exceed the
DataNode page cache, for example, approximately 2x DataNode memory.
- Because the corpus does not fit in main memory, the kernel evicts older
pages as newer blocks are written.
- By the time the measured phase re-reads a file, its pages are no longer
expected to be cached, causing the measured read to fall through to disk.
- The read workload spreads file selections across the entire corpus rather
than replaying a hot subset, keeping the page-cache hit rate near zero.
- This allows the tool to measure actual disk behavior rather than
page-cache performance.
- **Client-side latency and throughput metrics**
- Reports latency distributions for reads and writes:
- p50
- p75
- p95
- p99
- min
- max
- mean
- stddev
- Reports effective QPS and throughput for both reads and writes.
## Multi-Client Scale-Out
Aggregate load can be scaled by running the tool concurrently on multiple
client hosts against the same cluster.
The offered load is the sum of each client's configured throughput.
Multi-client usage and guidance are documented, including:
- Using distinct write/read directories per client.
- Sharing the same `favoredDataNodes` configuration.
- Running multiple instances concurrently.
## Configuration
Workload configuration is specified through a properties file.
Supported properties include:
| Property | Description |
|---|---|
| `favoredDataNodes` | DataNodes to target using HDFS favored-nodes hints |
| `testWriteDirectory` | Directory used for write workload |
| `testReadDirectories` | Directories containing the read corpus |
| `blockSizeMB` | HDFS block size and I/O unit |
| `testDurationSeconds` | Measured workload duration |
| `writeThroughputMB` | Target write throughput |
| `readThroughputMB` | Target read throughput |
| `testReadFileSizeGB` | Size of the read corpus |
| `preTestWriteThroughputMB` | Pre-test corpus write throughput |
| `preTestWriteDurationSeconds` | Maximum duration of the pre-test phase |
| `endWriteThroughputMB` | Optional final write throughput for ramping |
| `endReadThroughputMB` | Optional final read throughput for ramping |
Any property can be overridden using `-D` through `ToolRunner`.
## Documentation
Every configuration property, the run command, cold-read/page-cache
rationale, and multi-client scale-out guidance are documented in:
- The `HdfsStressTest` class Javadoc.
- The new user guide:
`hadoop-hdfs/src/site/markdown/HdfsStressTest.md`
- The HDFS site menu, which links to the new user guide.
### Run Example
```bash
hadoop jar hadoop-hdfs-<version>-tests.jar \
org.apache.hadoop.hdfs.HdfsStressTest \
/path/to/stress.properties
````
## Validation
`loadConfig` fails fast on misconfigurations that could otherwise produce
misleading results.
### Block Size Validation
`blockSizeMB` must be a positive number of MB.
The block size is used as the I/O unit, as well as a divisor when sizing the
corpus and calculating per-operation rates.
### Read Workload Validation
When the read workload is enabled:
```text
readThroughputMB > 0
```
`testReadFileSizeGB` must also be positive.
Reads are served only from the pre-test corpus. Without a valid corpus size,
the tool could silently run zero readers and still exit successfully.
### Cold-Read Corpus Validation
The pre-test phase creates a corpus large enough to exceed the DataNode page
cache so that subsequent reads fall through to disk.
The read workload distributes file selections across the entire corpus rather
than repeatedly reading a hot subset, keeping the page-cache hit rate near
zero.
If the pre-test is time-boxed using `preTestWriteDurationSeconds` and stops
before reaching `testReadFileSizeGB`, the tool prints a `WARNING`.
The warning indicates that some reads may be served from the page cache and
that the resulting numbers could therefore be optimistic, rather than
silently reporting warm reads as cold.
## Testing
### `TestHdfsStressTest`
Uses `MiniDFSCluster` to validate individual building blocks and the
end-to-end `Tool` execution.
The test verifies:
* `writeBlockFile`
* Produces an exactly block-sized file.
* Uses the configured replication factor and block size.
* Writes the expected payload.
* `readWholeFile`
* Reads a complete block through EOF.
* Pre-test phase
* Creates a block-sized cold-read corpus.
* Round-robins files across multiple directories.
* Honors the configured duration cap.
* Emits a warning to stderr when the capped corpus is smaller than the
requested read corpus size.
* Write-only workload
* Drives the write path through `ToolRunner`.
* Does not create a read corpus.
### `TestHdfsStressTestHelpers`
Provides fast, cluster-free unit tests for pacing, measurement, and
configuration helpers.
#### Rate Limiter
Tests:
* Token-bucket pacing and request spacing.
* Runtime rate changes.
#### Latency Statistics
Tests:
* Nanosecond-to-millisecond conversion.
* Latency sorting.
* Bounded-memory reservoir sampling.
#### Configuration Validation
Tests:
* `blockSizeMB` must be positive.
* Enabling the read workload requires a positive
`testReadFileSizeGB`.
* Valid read-only configurations load successfully.
* Valid write-only configurations load successfully.
These validations prevent configurations that would otherwise silently
perform
no reads or produce misleading benchmark results.
### How was this patch tested?
Tested by newly added unit test
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> HDFS-Test Add a compact standalone HDFS DataNode read/write stress tester
> -------------------------------------------------------------------------
>
> Key: HDFS-17977
> URL: https://issues.apache.org/jira/browse/HDFS-17977
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks, hdfs, test
> Reporter: Rajan Dhabalia
> Priority: Major
>
> h2. Summary
> Add a compact, single-process HDFS read/write stress tester that generates
> controlled throughput against targeted DataNodes and reports client-side
> latency distributions.
> h2. Description
> h3. Motivation
> `TestDFSIO` is the standard HDFS I/O benchmark but is not well suited for
> targeted DataNode stress testing.
> Key limitations:
> * Requires MapReduce/YARN.
> * Load is distributed through MapReduce scheduling rather than directly
> targeting DataNodes.
> * Limited control over steady throughput/QPS against selected DataNodes.
> * Primarily reports aggregate throughput rather than client-side p50/p95/p99
> latency.
> * Does not provide a reliable mechanism for cold-read workloads targeting
> disk I/O.
> `HdfsStressTest` provides a lightweight standalone tool for controlled
> DataNode performance and stress testing.
> h3. Approach
> Introduce `HdfsStressTest`, a single-process load generator that depends only
> on the HDFS client.
> Key capabilities:
> # *Controlled throughput*
> ## Global token-bucket rate limiter.
> ## Configurable read/write throughput.
> ## Optional linear throughput ramp.
> # *Targeted DataNodes*
> ## Use HDFS favored-nodes hints to direct writes to selected DataNodes.
> # *Configurable I/O size*
> ## Configurable block/file size.
> ## Block-sized operations for consistent latency measurements.
> # *Cold-read workload*
> ## Pre-create a read corpus before testing.
> ## Corpus can exceed DataNode page cache to reduce cache effects.
> # *Latency and throughput metrics*
> ## p50, p75, p95, p99, min, max, mean, and standard deviation.
> ## Effective QPS and throughput for reads/writes.
> # *Client-side scale-out*
> ## Multiple clients can run concurrently.
> ## Aggregate load is the sum of configured load across clients.
> Configuration is provided through a Java properties file, with individual
> properties optionally overridden using `-D` through `ToolRunner`.
> h3. Configuration
> ||Property||Default||Description||
> |`favoredDataNodes`|None|Comma-separated DataNode host:port list used as
> favored nodes for writes|
> |`replication`|3|Replication factor for generated files|
> |`blockSizeMB`|128|Block/file size for read/write operations|
> |`testWriteDirectory`|None|HDFS directory for write workload; omit to disable
> writes|
> |`writeThroughputMB`|0|Target write throughput; `0` disables writes|
> |`endWriteThroughputMB`|0|Optional end value for linear write-throughput ramp|
> |`writeThreads`|-1|Number of writer threads; `-1` uses automatic
> configuration|
> |`testReadDirectories`|None|HDFS directories for read workload/cold-read
> corpus|
> |`readThroughputMB`|0|Target read throughput; `0` disables reads|
> |`endReadThroughputMB`|0|Optional end value for linear read-throughput ramp|
> |`readThreads`|-1|Number of reader threads; `-1` uses automatic configuration|
> |`testReadFileSizeGB`|0|Size of pre-created cold-read corpus; `0` disables
> corpus generation|
> |`preTestWriteThroughputMB`|0|Optional throughput limit for corpus generation|
> |`preTestWriteDurationSeconds`|0|Optional time limit for corpus generation|
> |`testDurationSeconds`|60|Duration of measured workload|
> h3. Example
> {code:java}
> hadoop jar hadoop-hdfs-<version>-tests.jar \
> org.apache.hadoop.hdfs.HdfsStressTest \
> /path/to/stress.properties
> {code}
> h3. Results
> The stress tester provides:
> * Controlled read/write load against targeted DataNodes.
> * Reproducible workloads against specific replica sets.
> * Cold-read workloads for storage-path testing.
> * Client-side latency distributions and tail latency.
> * Effective throughput and QPS measurements.
> * Execution without MapReduce/YARN.
> * Scale-out through multiple client processes.
> The tool complements rather than replaces `TestDFSIO`.
> h3. Comparison with TestDFSIO
> ||Dimension||TestDFSIO||HdfsStressTest||
> |Target specific DataNodes|Limited|Supported through favored-nodes hints|
> |Controlled throughput|Limited by MapReduce|Explicit control|
> |Throughput ramp|No|Supported|
> |Cold-read workload|Not specifically supported|Supported|
> |Client-side latency|Aggregate metrics|p50/p75/p95/p99 and other statistics|
> |External dependencies|MapReduce/YARN|HDFS client only|
> |Targeted DataNode stress|Difficult|Primary use case|
> |Scale-out|MapReduce-based|Multiple standalone clients|
> |Single-process execution|No|Yes|
> h3. Expected Impact
> Improves reproducibility and control for DataNode performance testing through:
> * Precise offered-load control.
> * Targeted DataNode/replica-set testing.
> * Repeatable cold-read workloads.
> * Client-side tail-latency measurements.
> * Lightweight execution without MapReduce/YARN.
> * Easy scale-out using multiple client processes.
> The primary benefit is {*}better observability and control during DataNode
> performance and stress testing{*}, rather than production throughput
> improvement.
> h2. Robustness and Input Validation
> The tool validates configuration at startup to prevent misleading benchmark
> results:
> * `blockSizeMB` must be positive.
> * Read workloads (`readThroughputMB > 0`) require a positive
> `testReadFileSizeGB`.
> * If `preTestWriteDurationSeconds` limits corpus generation before the
> requested `testReadFileSizeGB` is reached, the tool prints a *WARNING* that
> reads may be served from OS page cache and recommends increasing or removing
> the time limit.
> h2. Testing
> `TestHdfsStressTest` (MiniDFSCluster) covers:
> * Block-sized file creation with configured replication and payload.
> * Full-file reads to EOF.
> * Pre-test cold-read corpus generation across multiple directories.
> * Corpus generation duration limits and cache warnings.
> * Write-only execution through `ToolRunner`.
> `TestHdfsStressTestHelpers` provides cluster-free unit tests for:
> * Token-bucket rate limiting and pacing.
> * Runtime rate changes.
> * Latency conversion and sorting.
> * Bounded-memory reservoir sampling.
> * Configuration validation.
> * Valid read-only and write-only configurations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
