[
https://issues.apache.org/jira/browse/HDFS-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098421#comment-18098421
]
ASF GitHub Bot commented on HDFS-17953:
---------------------------------------
singer-bin opened a new pull request, #8626:
URL: https://github.com/apache/hadoop/pull/8626
### Description of PR
JIRA: [HDFS-17953](https://issues.apache.org/jira/browse/HDFS-17953)
#### Problem
When a DataNode is configured with HA NameNodes (`dfs.ha.namenodes.<nsId>`),
but
one of the configured NameNode IDs has no (or a wrong) RPC address — or the
NameNode is permanently unreachable — the
`IncrementalBlockReportManager.pendingIBRs` map for that actor grows without
bound and eventually causes the DataNode to OOM.
In `BPOfferService#notifyNamenodeBlock`, every block receive/delete event is
added to the IBR queue of *every* `BPServiceActor`, including the actor for
the
unreachable NameNode. In `IncrementalBlockReportManager#sendIBRs`, on send
failure `putMissing()` puts all blocks back into the queue. When the
NameNode is
permanently unreachable the send always fails, new events keep arriving, and
the
queue grows indefinitely — there is no size limit, no TTL and no eviction.
The only existing protection (HDFS-9917) calls `clearIBRs()` during
`reRegister()` for STANDBY/OBSERVER NNs, but `reRegister()` is never
triggered
when the NN is completely unreachable (no `DNA_REGISTER` command can be
received). A production heap dump showed ~223M `ReceivedDeletedBlockInfo`
instances occupying ~30 GB, caused by a stale `nn3` whose RPC address was
never
configured (it fell back to the logical `nameservice:8020` which never
resolves).
#### Fix
1. Add a configurable cap `dfs.datanode.ibr.max.pending.size` (default
1,000,000). When exceeded, the pending IBR queue is cleared with a
warning.
2. Add a staleness guard `dfs.datanode.ibr.max.stale.interval.ms` (default
30 min): if no successful IBR send happens within this window while
entries
are pending, the queue is cleared.
3. After clearing (overflow or staleness), a Full Block Report is scheduled
so
the NameNode gets a complete, consistent view once reachable again.
This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an
optimization for incremental updates between FBRs. Dropping queued IBRs for
an
unreachable NN cannot cause inconsistency because a fresh FBR is sent on
reconnect/re-register.
Both protections can be disabled by setting the corresponding value to `0`.
#### How was this patch tested?
New unit tests in `TestIncrementalBlockReportManager`:
- `testIBRQueueSizeLimit` — queue is capped and cleared on overflow.
- `testIBRQueueStaleInterval` — queue is cleared after the stale interval.
- `testIBRQueueNoLimit` — cap disabled (0) keeps old behavior.
- `testIBRDeduplication` — per-block dedup still works.
- `testTotalPendingIBRSizeMultipleStorages` — size counting across storages.
#### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: N/A
- [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files? N/A
> DataNode IBR pendingIBRs grows unbounded when a NameNode is unreachable,
> causing OOM
> ------------------------------------------------------------------------------------
>
> Key: HDFS-17953
> URL: https://issues.apache.org/jira/browse/HDFS-17953
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.4.3
> Reporter: yanbin.zhang
> Assignee: yanbin.zhang
> Priority: Major
>
> h2. Problem
> When a DataNode is configured with HA NameNodes
> (\{{dfs.ha.namenodes.<nsId>}}),
> but one of the configured NameNode IDs has no (or a wrong) RPC address — or
> the
> NameNode is permanently unreachable — the
> {\{IncrementalBlockReportManager.pendingIBRs}} map for that actor grows
> without
> bound and eventually causes the DataNode to OOM.
> h2. Root Cause
> In \{{BPOfferService#notifyNamenodeBlock}}, every block receive/delete event
> is
> added to the IBR queue of *every* BPServiceActor, including the actor for the
> unreachable NameNode. In \{{IncrementalBlockReportManager#sendIBRs}}:
> # \{{generateIBRs()}} drains the pending blocks from the queue.
> # \{{namenode.blockReceivedAndDeleted()}} attempts the RPC.
> # on failure, \{{putMissing()}} puts all the blocks *back* into the queue.
> When the NameNode is permanently unreachable:
> * \{{sendIBRs()}} always fails, so blocks are always put back;
> * new block events keep being added via \{{addRDBI()}};
> * \{{pendingIBRs}} grows indefinitely — there is *no size limit, no TTL and no
> eviction*.
> The only existing protection (HDFS-9917) calls \{{clearIBRs()}} during
> {\{reRegister()}} for STANDBY/OBSERVER NNs, but \{{reRegister()}} is never
> triggered when the NN is completely unreachable (no \{{DNA_REGISTER}} command
> can
> be received).
> h2. How to reproduce
> Configure HA with 3 NameNode IDs but only provide RPC addresses for 2:
> {code:xml}
> <property>
> <name>dfs.ha.namenodes.mycluster</name>
> <value>nn1,nn2,nn3</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn1</name>
> <value>host1:8020</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn2</name>
> <value>host2:8020</value>
> </property>
> <!-- nn3 RPC address is missing; it falls back to the logical
> nameservice id "mycluster:8020", which never resolves -->
> {code}
> The DataNode creates a BPServiceActor for \{{mycluster:8020}} that can never
> connect. The DataNode log fills with:
> {noformat}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to
> server: mycluster:8020
> {noformat}
> After running for some time the DataNode heap is dominated by
> {\{ReceivedDeletedBlockInfo}} / \{{Block}} / \{{HashMap$Node}} entries
> belonging to
> that dead actor's IBR queue. A production heap dump showed ~223M
> {\{ReceivedDeletedBlockInfo}} instances occupying ~30 GB.
> h2. Impact
> * DataNode OOM / crash.
> * After a restart the heap starts filling again immediately, because the
> configuration is unchanged.
> * Affects any cluster where a NameNode is permanently removed/decommissioned
> without updating \{{dfs.ha.namenodes.<nsId>}}, or where an RPC address is
> simply misconfigured.
> h2. Proposed fix
> # Add a configurable cap on the pending IBR queue,
> {\{dfs.datanode.ibr.max.pending.size}} (default 1,000,000). When exceeded, the
> queue is cleared and a warning is logged.
> # Add a staleness guard \{{dfs.datanode.ibr.max.stale.interval.ms}}
> (default 30 min): if there has been no successful IBR send within this window
> while entries are pending, the queue is cleared.
> # After clearing (overflow or staleness), schedule a Full Block Report so the
> NameNode gets a complete, consistent view once it is reachable again.
> This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an
> optimization for incremental updates between FBRs. Dropping queued IBRs for an
> unreachable NN cannot cause inconsistency because a fresh FBR is sent on
> reconnect/re-register.
> h2. Test plan
> New unit tests in \{{TestIncrementalBlockReportManager}}:
> * \{{testIBRQueueSizeLimit}} — queue is capped and cleared on overflow.
> * \{{testIBRQueueStaleInterval}} — queue is cleared after the stale interval.
> * \{{testIBRQueueNoLimit}} — cap disabled (0) keeps old behavior.
> * \{{testIBRDeduplication}} — per-block dedup still works.
> * \{{testTotalPendingIBRSizeMultipleStorages}} — size counting across
> storages.
> ---
> ## Labels (可选)
> datanode, ibr, memory-leak, oom
> ## 关联关系
> - Relates to / Is related to: **HDFS-9917**, **HDFS-15113**
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
