Rajan Dhabalia created HDFS-17976:
-------------------------------------
Summary: HDFS DataNode add configurable inactivity-based timeout
for DataNode block transfers
Key: HDFS-17976
URL: https://issues.apache.org/jira/browse/HDFS-17976
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Reporter: Rajan Dhabalia
h2. Summary
Add a configurable inactivity-based timeout to safely terminate stalled
DataNode block transfers, reclaim resources, and preserve partially written
replicas for recovery.
h2. Issue Type
Improvement
h2. Component/s
datanode
h2. Description
h3. Motivation
A block write remains active until the client sends the final packet and the
block is finalized. If a client crashes, loses connectivity, or becomes
unresponsive, the DataNode transfer thread can remain blocked waiting for
additional data.
Stalled transfers can accumulate over time, consuming transfer threads and
other resources and potentially affecting healthy client operations.
h3. Approach
Introduce a configurable inactivity timeout for DataNode block transfers.
When enabled:
* Track packet-arrival activity for each ongoing block write.
* Schedule an inactivity check for each active transfer.
* Terminate the transfer if no packet is received within the configured
timeout.
* Flush buffered data before termination.
* Close the input stream to interrupt blocked reads.
* Keep the replica in *RBW (Replica Being Written)* state.
* Allow existing NameNode lease recovery and block synchronization to recover
and finalize the replica.
The timeout uses a shared `ScheduledExecutorService` and is created lazily only
when enabled. When disabled, there are no additional scheduler threads or
timeout processing.
h3. Configuration
||Property||Default||Description||
|`dfs.datanode.last.packet.receive.timeout.ms`|0|Inactivity timeout for an
ongoing block transfer. If no packet is received within the configured window,
the transfer is considered stalled and terminated. `0` or negative disables the
feature.|
h3. Transfer Lifecycle
{code:java}
Client starts block write
|
v
DataNode receives packets
|
v
Track packet activity
|
v
Packet received?
/ \
Yes No
| |
v v
Reset timer Timeout reached
|
v
Flush buffered data
|
v
Close input stream
|
v
Replica remains RBW
|
v
Existing lease recovery /
block synchronization
|
v
Block finalized
{code}
h3. Data Safety
When a stalled transfer is terminated:
# Buffered data is flushed.
# The input stream is closed.
# The replica remains in RBW state.
# Existing HDFS recovery mechanisms handle subsequent recovery/finalization.
This preserves data already received by the DataNode while reclaiming stalled
transfer resources.
h3. Results
The inactivity timeout provides:
* Bounded resource usage for stalled transfers.
* Prevention of indefinite transfer-thread retention after client failures.
* Reduced risk of transfer-thread exhaustion during client failure bursts.
* Preservation of partially written replica data.
* Compatibility with existing HDFS recovery mechanisms.
* No additional overhead when disabled.
h3. Estimated Impact
This is primarily a reliability and resource-reclamation improvement rather
than a throughput optimization.
||Dimension||Without Timeout||With Inactivity Timeout||
|Stalled transfer lifetime|Potentially unbounded|Bounded by configured timeout|
|Transfer threads|Can accumulate|Reclaimed after inactivity|
|Client failure bursts|Risk of resource exhaustion|Resource usage remains
bounded|
|Partially written data|Preserved through existing recovery|Preserved; replica
remains RBW|
|Normal transfers|Existing behavior|Unchanged|
|Disabled overhead|Existing behavior|No additional scheduler/timeout processing|
h3. Operational Considerations
The timeout should be configured based on workload characteristics. An overly
aggressive timeout could terminate legitimately slow transfers, while a
sufficiently large timeout allows temporary network or client stalls while
still reclaiming resources from genuinely stalled transfers.
The feature is therefore *disabled by default* and can be enabled explicitly by
operators.
h2. Expected Impact
* Improve DataNode resilience to crashed, disconnected, or hung clients.
* Prevent stalled transfers from holding resources indefinitely.
* Bound transfer-resource usage during client failure bursts.
* Preserve partially written data for existing recovery mechanisms.
* Keep normal block-transfer behavior unchanged when disabled.
h2. Backward Compatibility
The feature is {*}disabled by default{*}. Existing DataNode block-transfer
behavior remains unchanged unless `dfs.datanode.last.packet.receive.timeout.ms`
is explicitly configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
