[
https://issues.apache.org/jira/browse/HDFS-17976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111766#comment-18111766
]
ASF GitHub Bot commented on HDFS-17976:
---------------------------------------
rdhabalia opened a new pull request, #8718:
URL: https://github.com/apache/hadoop/pull/8718
### Description of PR
A block written to a DataNode remains active until the client sends the final
packet and the block is finalized.
When a client crashes, loses network connectivity, or hangs before sending
the
last packet, the DataNode transfer thread can remain blocked indefinitely
while
waiting for input. Over time, these stalled transfers can accumulate and
cause:
- Transfer-thread starvation
- Resource leakage
- Degraded DataNode responsiveness
Today, there is no built-in inactivity detection to reclaim resources from
stalled transfers. The system relies on client behavior or eventual lease
recovery, which may occur long after the actual failure.
## Solution
Introduce a configurable, inactivity-based timeout for block transfers on the
DataNode.
When enabled, the DataNode tracks packet-arrival activity for each ongoing
block write using a monotonic clock. A single scheduled check per transfer
runs every `timeout / 2` and compares the time since the last received packet
against `timeout / 2`.
### Transfer Activity Detection
- If a packet arrived within `timeout / 2`:
- The stream is considered active.
- The check is rescheduled.
- This provides an automatic reset for healthy streams.
- Otherwise:
- The transfer is considered stalled.
- The transfer is aborted.
Polling every `timeout / 2` bounds the detection latency to between
`timeout / 2` and `timeout` after the client stops sending.
### Safe Transfer Abort
The abort runs on the shared scheduler thread and deliberately performs **no
disk I/O**.
The on-disk streams (`out` / `checksumOut`) are written by the receive thread
**without holding the `BlockReceiver` monitor**. Flushing these streams from
the scheduler thread could race with those writes and potentially corrupt the
replica.
Instead, the scheduler:
1. Closes only the client input stream.
2. This unblocks the receive thread's blocked socket read.
3. The receive thread unwinds and performs its own single-threaded cleanup.
Data already acknowledged to the client is already durable on disk. The
replica remains in the **RBW (Replica Being Written)** state, allowing the
NameNode's existing lease-recovery / block-synchronization path to finalize
the block without data loss.
## Scheduler Design
The timeout checks use a shared `ScheduledThreadPoolExecutor` with **2 daemon
threads**.
The scheduler:
- Is created lazily at DataNode startup only when the timeout is configured.
- Adds **zero threads and zero overhead when the feature is disabled**.
- Uses `remove-on-cancel` so per-block checks cancelled when a transfer
completes do not accumulate in the delay queue.
- Does not execute delayed tasks after shutdown.
The last-packet timestamp is updated only when the timeout feature is
enabled,
avoiding unnecessary atomic writes on the hot path when the feature is
disabled.
## Client Socket Timeout Considerations
Because a stall is declared after no packet is received for `timeout / 2`,
the DataNode timeout must be configured comfortably larger than the client
socket read timeout.
A healthy idle `hflush` / `hsync` stream still sends heartbeat packets
roughly
every half of the socket timeout.
The DataNode logs a warning if the configured transfer timeout is not
sufficiently larger than the client socket timeout.
## Configuration
| Configuration | Default | Description |
|---|---:|---|
| `dfs.datanode.last.packet.receive.timeout.ms` | `0` | Last-packet
inactivity timeout in milliseconds. `0` disables the feature. |
A typical enabled value is:
```text
dfs.datanode.last.packet.receive.timeout.ms=600000
````
This corresponds to a **10-minute timeout**.
The change is fully backward compatible and disabled by default.
## Testing
### `TestBlockReceiverLastPacketTimeout`
Verifies the DataNode-side scheduler wiring:
* The shared scheduler is created only when the timeout is enabled.
* `remove-on-cancel` is configured correctly.
* Delayed tasks do not execute after scheduler shutdown.
### `TestBlockReceiverTransferTimeout`
Runs an end-to-end test using `MiniDFSCluster` and verifies that a genuinely
stuck client is deterministically aborted.
The test uses a single DataNode with DataNode replacement disabled so that a
broken pipeline cannot be silently recovered.
It also verifies that healthy streams are never falsely aborted, including:
* Full transfers
* Slow-but-steady writers
* Writers whose packet gaps remain just below the timeout threshold, even
when the total transfer time exceeds the timeout
* Disabled timeout configuration
* Multiple concurrent transfers
### How was this patch tested?
Tested by newly added unit test
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> HDFS DataNode add configurable inactivity-based timeout for DataNode block
> transfers
> ------------------------------------------------------------------------------------
>
> Key: HDFS-17976
> URL: https://issues.apache.org/jira/browse/HDFS-17976
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Reporter: Rajan Dhabalia
> Priority: Major
>
> h2. Summary
> Add a configurable inactivity-based timeout to safely terminate stalled
> DataNode block transfers, reclaim resources, and preserve partially written
> replicas for recovery.
> h2. Issue Type
> Improvement
> h2. Component/s
> datanode
> h2. Description
> h3. Motivation
> A block write remains active until the client sends the final packet and the
> block is finalized. If a client crashes, loses connectivity, or becomes
> unresponsive, the DataNode transfer thread can remain blocked waiting for
> additional data.
> Stalled transfers can accumulate over time, consuming transfer threads and
> other resources and potentially affecting healthy client operations.
> h3. Approach
> Introduce a configurable inactivity timeout for DataNode block transfers.
> When enabled:
> * Track packet-arrival activity for each ongoing block write.
> * Schedule an inactivity check for each active transfer.
> * Terminate the transfer if no packet is received within the configured
> timeout.
> * Flush buffered data before termination.
> * Close the input stream to interrupt blocked reads.
> * Keep the replica in *RBW (Replica Being Written)* state.
> * Allow existing NameNode lease recovery and block synchronization to
> recover and finalize the replica.
> The timeout uses a shared `ScheduledExecutorService` and is created lazily
> only when enabled. When disabled, there are no additional scheduler threads
> or timeout processing.
> h3. Configuration
> ||Property||Default||Description||
> |`dfs.datanode.last.packet.receive.timeout.ms`|0|Inactivity timeout for an
> ongoing block transfer. If no packet is received within the configured
> window, the transfer is considered stalled and terminated. `0` or negative
> disables the feature.|
> h3. Transfer Lifecycle
> {code:java}
> Client starts block write
> |
> v
> DataNode receives packets
> |
> v
> Track packet activity
> |
> v
> Packet received?
> / \
> Yes No
> | |
> v v
> Reset timer Timeout reached
> |
> v
> Flush buffered data
> |
> v
> Close input stream
> |
> v
> Replica remains RBW
> |
> v
> Existing lease recovery /
> block synchronization
> |
> v
> Block finalized
> {code}
> h3. Data Safety
> When a stalled transfer is terminated:
> # Buffered data is flushed.
> # The input stream is closed.
> # The replica remains in RBW state.
> # Existing HDFS recovery mechanisms handle subsequent recovery/finalization.
> This preserves data already received by the DataNode while reclaiming stalled
> transfer resources.
> h3. Results
> The inactivity timeout provides:
> * Bounded resource usage for stalled transfers.
> * Prevention of indefinite transfer-thread retention after client failures.
> * Reduced risk of transfer-thread exhaustion during client failure bursts.
> * Preservation of partially written replica data.
> * Compatibility with existing HDFS recovery mechanisms.
> * No additional overhead when disabled.
> h3. Estimated Impact
> This is primarily a reliability and resource-reclamation improvement rather
> than a throughput optimization.
> ||Dimension||Without Timeout||With Inactivity Timeout||
> |Stalled transfer lifetime|Potentially unbounded|Bounded by configured
> timeout|
> |Transfer threads|Can accumulate|Reclaimed after inactivity|
> |Client failure bursts|Risk of resource exhaustion|Resource usage remains
> bounded|
> |Partially written data|Preserved through existing recovery|Preserved;
> replica remains RBW|
> |Normal transfers|Existing behavior|Unchanged|
> |Disabled overhead|Existing behavior|No additional scheduler/timeout
> processing|
> h3. Operational Considerations
> The timeout should be configured based on workload characteristics. An overly
> aggressive timeout could terminate legitimately slow transfers, while a
> sufficiently large timeout allows temporary network or client stalls while
> still reclaiming resources from genuinely stalled transfers.
> The feature is therefore *disabled by default* and can be enabled explicitly
> by operators.
> h2. Expected Impact
> * Improve DataNode resilience to crashed, disconnected, or hung clients.
> * Prevent stalled transfers from holding resources indefinitely.
> * Bound transfer-resource usage during client failure bursts.
> * Preserve partially written data for existing recovery mechanisms.
> * Keep normal block-transfer behavior unchanged when disabled.
> h2. Backward Compatibility
> The feature is {*}disabled by default{*}. Existing DataNode block-transfer
> behavior remains unchanged unless
> `dfs.datanode.last.packet.receive.timeout.ms` is explicitly configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
