[
https://issues.apache.org/jira/browse/HDFS-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17977:
----------------------------------
Labels: pull-request-available (was: )
> HDFS-Test Add a compact standalone HDFS DataNode read/write stress tester
> -------------------------------------------------------------------------
>
> Key: HDFS-17977
> URL: https://issues.apache.org/jira/browse/HDFS-17977
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks, hdfs, test
> Reporter: Rajan Dhabalia
> Priority: Major
> Labels: pull-request-available
>
> h2. Summary
> Add a compact, single-process HDFS read/write stress tester that generates
> controlled throughput against targeted DataNodes and reports client-side
> latency distributions.
> h2. Description
> h3. Motivation
> `TestDFSIO` is the standard HDFS I/O benchmark but is not well suited for
> targeted DataNode stress testing.
> Key limitations:
> * Requires MapReduce/YARN.
> * Load is distributed through MapReduce scheduling rather than directly
> targeting DataNodes.
> * Limited control over steady throughput/QPS against selected DataNodes.
> * Primarily reports aggregate throughput rather than client-side p50/p95/p99
> latency.
> * Does not provide a reliable mechanism for cold-read workloads targeting
> disk I/O.
> `HdfsStressTest` provides a lightweight standalone tool for controlled
> DataNode performance and stress testing.
> h3. Approach
> Introduce `HdfsStressTest`, a single-process load generator that depends only
> on the HDFS client.
> Key capabilities:
> # *Controlled throughput*
> ## Global token-bucket rate limiter.
> ## Configurable read/write throughput.
> ## Optional linear throughput ramp.
> # *Targeted DataNodes*
> ## Use HDFS favored-nodes hints to direct writes to selected DataNodes.
> # *Configurable I/O size*
> ## Configurable block/file size.
> ## Block-sized operations for consistent latency measurements.
> # *Cold-read workload*
> ## Pre-create a read corpus before testing.
> ## Corpus can exceed DataNode page cache to reduce cache effects.
> # *Latency and throughput metrics*
> ## p50, p75, p95, p99, min, max, mean, and standard deviation.
> ## Effective QPS and throughput for reads/writes.
> # *Client-side scale-out*
> ## Multiple clients can run concurrently.
> ## Aggregate load is the sum of configured load across clients.
> Configuration is provided through a Java properties file, with individual
> properties optionally overridden using `-D` through `ToolRunner`.
> h3. Configuration
> ||Property||Default||Description||
> |`favoredDataNodes`|None|Comma-separated DataNode host:port list used as
> favored nodes for writes|
> |`replication`|3|Replication factor for generated files|
> |`blockSizeMB`|128|Block/file size for read/write operations|
> |`testWriteDirectory`|None|HDFS directory for write workload; omit to disable
> writes|
> |`writeThroughputMB`|0|Target write throughput; `0` disables writes|
> |`endWriteThroughputMB`|0|Optional end value for linear write-throughput ramp|
> |`writeThreads`|-1|Number of writer threads; `-1` uses automatic
> configuration|
> |`testReadDirectories`|None|HDFS directories for read workload/cold-read
> corpus|
> |`readThroughputMB`|0|Target read throughput; `0` disables reads|
> |`endReadThroughputMB`|0|Optional end value for linear read-throughput ramp|
> |`readThreads`|-1|Number of reader threads; `-1` uses automatic configuration|
> |`testReadFileSizeGB`|0|Size of pre-created cold-read corpus; `0` disables
> corpus generation|
> |`preTestWriteThroughputMB`|0|Optional throughput limit for corpus generation|
> |`preTestWriteDurationSeconds`|0|Optional time limit for corpus generation|
> |`testDurationSeconds`|60|Duration of measured workload|
> h3. Example
> {code:java}
> hadoop jar hadoop-hdfs-<version>-tests.jar \
> org.apache.hadoop.hdfs.HdfsStressTest \
> /path/to/stress.properties
> {code}
> h3. Results
> The stress tester provides:
> * Controlled read/write load against targeted DataNodes.
> * Reproducible workloads against specific replica sets.
> * Cold-read workloads for storage-path testing.
> * Client-side latency distributions and tail latency.
> * Effective throughput and QPS measurements.
> * Execution without MapReduce/YARN.
> * Scale-out through multiple client processes.
> The tool complements rather than replaces `TestDFSIO`.
> h3. Comparison with TestDFSIO
> ||Dimension||TestDFSIO||HdfsStressTest||
> |Target specific DataNodes|Limited|Supported through favored-nodes hints|
> |Controlled throughput|Limited by MapReduce|Explicit control|
> |Throughput ramp|No|Supported|
> |Cold-read workload|Not specifically supported|Supported|
> |Client-side latency|Aggregate metrics|p50/p75/p95/p99 and other statistics|
> |External dependencies|MapReduce/YARN|HDFS client only|
> |Targeted DataNode stress|Difficult|Primary use case|
> |Scale-out|MapReduce-based|Multiple standalone clients|
> |Single-process execution|No|Yes|
> h3. Expected Impact
> Improves reproducibility and control for DataNode performance testing through:
> * Precise offered-load control.
> * Targeted DataNode/replica-set testing.
> * Repeatable cold-read workloads.
> * Client-side tail-latency measurements.
> * Lightweight execution without MapReduce/YARN.
> * Easy scale-out using multiple client processes.
> The primary benefit is {*}better observability and control during DataNode
> performance and stress testing{*}, rather than production throughput
> improvement.
> h2. Robustness and Input Validation
> The tool validates configuration at startup to prevent misleading benchmark
> results:
> * `blockSizeMB` must be positive.
> * Read workloads (`readThroughputMB > 0`) require a positive
> `testReadFileSizeGB`.
> * If `preTestWriteDurationSeconds` limits corpus generation before the
> requested `testReadFileSizeGB` is reached, the tool prints a *WARNING* that
> reads may be served from OS page cache and recommends increasing or removing
> the time limit.
> h2. Testing
> `TestHdfsStressTest` (MiniDFSCluster) covers:
> * Block-sized file creation with configured replication and payload.
> * Full-file reads to EOF.
> * Pre-test cold-read corpus generation across multiple directories.
> * Corpus generation duration limits and cache warnings.
> * Write-only execution through `ToolRunner`.
> `TestHdfsStressTestHelpers` provides cluster-free unit tests for:
> * Token-bucket rate limiting and pacing.
> * Runtime rate changes.
> * Latency conversion and sorting.
> * Bounded-memory reservoir sampling.
> * Configuration validation.
> * Valid read-only and write-only configurations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
