[
https://issues.apache.org/jira/browse/HDFS-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098772#comment-18098772
]
ASF GitHub Bot commented on HDFS-17953:
---------------------------------------
singer-bin commented on PR #8626:
URL: https://github.com/apache/hadoop/pull/8626#issuecomment-5067448567
Thanks for the review, @Hexiaoqiao ! You are right — the pending-block-count
cap alone is enough to bound the DataNode heap footprint, so
`maxStaleIntervalMs` was redundant. I have removed it and the related config
(`dfs.datanode.ibr.max.stale.interval.ms`).
The fix now keeps a single, simple guard:
`dfs.datanode.ibr.max.pending.blocks` (default 1,000,000). The pending block
count is tracked in O(1), and when it reaches the cap
`IncrementalBlockReportManager#clearIBRsIfNeeded()` clears the queue and the
`BPServiceActor` schedules a full block report so the NameNode can resync once
it is reachable again. Set the value to 0 to disable.
Updated tests (all passing locally):
- `TestIncrementalBlockReportManager`: counter accounting, dedup, size-cap
clear, and cap-disabled behavior.
- `TestIncrementalBlockReports#testIBRQueueBoundedWhenNNUnreachable`:
end-to-end test with a MiniDFSCluster and an unreachable NN, asserting the
pending block count never exceeds the cap.
Please take another look. Thanks!
> DataNode IBR pendingIBRs grows unbounded when a NameNode is unreachable,
> causing OOM
> ------------------------------------------------------------------------------------
>
> Key: HDFS-17953
> URL: https://issues.apache.org/jira/browse/HDFS-17953
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.4.3
> Reporter: yanbin.zhang
> Assignee: yanbin.zhang
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> When a DataNode is configured with HA NameNodes
> (\{{dfs.ha.namenodes.<nsId>}}),
> but one of the configured NameNode IDs has no (or a wrong) RPC address — or
> the
> NameNode is permanently unreachable — the
> {\{IncrementalBlockReportManager.pendingIBRs}} map for that actor grows
> without
> bound and eventually causes the DataNode to OOM.
> h2. Root Cause
> In \{{BPOfferService#notifyNamenodeBlock}}, every block receive/delete event
> is
> added to the IBR queue of *every* BPServiceActor, including the actor for the
> unreachable NameNode. In \{{IncrementalBlockReportManager#sendIBRs}}:
> # \{{generateIBRs()}} drains the pending blocks from the queue.
> # \{{namenode.blockReceivedAndDeleted()}} attempts the RPC.
> # on failure, \{{putMissing()}} puts all the blocks *back* into the queue.
> When the NameNode is permanently unreachable:
> * \{{sendIBRs()}} always fails, so blocks are always put back;
> * new block events keep being added via \{{addRDBI()}};
> * \{{pendingIBRs}} grows indefinitely — there is *no size limit, no TTL and no
> eviction*.
> The only existing protection (HDFS-9917) calls \{{clearIBRs()}} during
> {\{reRegister()}} for STANDBY/OBSERVER NNs, but \{{reRegister()}} is never
> triggered when the NN is completely unreachable (no \{{DNA_REGISTER}} command
> can
> be received).
> h2. How to reproduce
> Configure HA with 3 NameNode IDs but only provide RPC addresses for 2:
> {code:xml}
> <property>
> <name>dfs.ha.namenodes.mycluster</name>
> <value>nn1,nn2,nn3</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn1</name>
> <value>host1:8020</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn2</name>
> <value>host2:8020</value>
> </property>
> <!-- nn3 RPC address is missing; it falls back to the logical
> nameservice id "mycluster:8020", which never resolves -->
> {code}
> The DataNode creates a BPServiceActor for \{{mycluster:8020}} that can never
> connect. The DataNode log fills with:
> {noformat}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to
> server: mycluster:8020
> {noformat}
> After running for some time the DataNode heap is dominated by
> {\{ReceivedDeletedBlockInfo}} / \{{Block}} / \{{HashMap$Node}} entries
> belonging to
> that dead actor's IBR queue. A production heap dump showed ~223M
> {\{ReceivedDeletedBlockInfo}} instances occupying ~30 GB.
> h2. Impact
> * DataNode OOM / crash.
> * After a restart the heap starts filling again immediately, because the
> configuration is unchanged.
> * Affects any cluster where a NameNode is permanently removed/decommissioned
> without updating \{{dfs.ha.namenodes.<nsId>}}, or where an RPC address is
> simply misconfigured.
> h2. Proposed fix
> # Add a configurable cap on the pending IBR queue,
> {\{dfs.datanode.ibr.max.pending.size}} (default 1,000,000). When exceeded, the
> queue is cleared and a warning is logged.
> # Add a staleness guard \{{dfs.datanode.ibr.max.stale.interval.ms}}
> (default 30 min): if there has been no successful IBR send within this window
> while entries are pending, the queue is cleared.
> # After clearing (overflow or staleness), schedule a Full Block Report so the
> NameNode gets a complete, consistent view once it is reachable again.
> This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an
> optimization for incremental updates between FBRs. Dropping queued IBRs for an
> unreachable NN cannot cause inconsistency because a fresh FBR is sent on
> reconnect/re-register.
> h2. Test plan
> New unit tests in \{{TestIncrementalBlockReportManager}}:
> * \{{testIBRQueueSizeLimit}} — queue is capped and cleared on overflow.
> * \{{testIBRQueueStaleInterval}} — queue is cleared after the stale interval.
> * \{{testIBRQueueNoLimit}} — cap disabled (0) keeps old behavior.
> * \{{testIBRDeduplication}} — per-block dedup still works.
> * \{{testTotalPendingIBRSizeMultipleStorages}} — size counting across
> storages.
> ---
> ## Labels (可选)
> datanode, ibr, memory-leak, oom
> ## 关联关系
> - Relates to / Is related to: **HDFS-9917**, **HDFS-15113**
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
