[
https://issues.apache.org/jira/browse/HDFS-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098497#comment-18098497
]
ASF GitHub Bot commented on HDFS-17953:
---------------------------------------
hadoop-yetus commented on PR #8626:
URL: https://github.com/apache/hadoop/pull/8626#issuecomment-5059404880
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 26m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 45s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 45s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8626/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 259 unchanged
- 0 fixed = 261 total (was 259) |
| +1 :green_heart: | mvnsite | 0m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 50s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 177m 19s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 253m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8626/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/8626 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux c5d6c20ef4c9 5.15.0-181-generic #191-Ubuntu SMP Fri May 22
19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 93ebc4ba74f0c18ef1c767c5f14ee39a9cc725f1 |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions |
/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
/usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8626/1/testReport/ |
| Max. process+thread count | 4415 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8626/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
> DataNode IBR pendingIBRs grows unbounded when a NameNode is unreachable,
> causing OOM
> ------------------------------------------------------------------------------------
>
> Key: HDFS-17953
> URL: https://issues.apache.org/jira/browse/HDFS-17953
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.4.3
> Reporter: yanbin.zhang
> Assignee: yanbin.zhang
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> When a DataNode is configured with HA NameNodes
> (\{{dfs.ha.namenodes.<nsId>}}),
> but one of the configured NameNode IDs has no (or a wrong) RPC address — or
> the
> NameNode is permanently unreachable — the
> {\{IncrementalBlockReportManager.pendingIBRs}} map for that actor grows
> without
> bound and eventually causes the DataNode to OOM.
> h2. Root Cause
> In \{{BPOfferService#notifyNamenodeBlock}}, every block receive/delete event
> is
> added to the IBR queue of *every* BPServiceActor, including the actor for the
> unreachable NameNode. In \{{IncrementalBlockReportManager#sendIBRs}}:
> # \{{generateIBRs()}} drains the pending blocks from the queue.
> # \{{namenode.blockReceivedAndDeleted()}} attempts the RPC.
> # on failure, \{{putMissing()}} puts all the blocks *back* into the queue.
> When the NameNode is permanently unreachable:
> * \{{sendIBRs()}} always fails, so blocks are always put back;
> * new block events keep being added via \{{addRDBI()}};
> * \{{pendingIBRs}} grows indefinitely — there is *no size limit, no TTL and no
> eviction*.
> The only existing protection (HDFS-9917) calls \{{clearIBRs()}} during
> {\{reRegister()}} for STANDBY/OBSERVER NNs, but \{{reRegister()}} is never
> triggered when the NN is completely unreachable (no \{{DNA_REGISTER}} command
> can
> be received).
> h2. How to reproduce
> Configure HA with 3 NameNode IDs but only provide RPC addresses for 2:
> {code:xml}
> <property>
> <name>dfs.ha.namenodes.mycluster</name>
> <value>nn1,nn2,nn3</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn1</name>
> <value>host1:8020</value>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.mycluster.nn2</name>
> <value>host2:8020</value>
> </property>
> <!-- nn3 RPC address is missing; it falls back to the logical
> nameservice id "mycluster:8020", which never resolves -->
> {code}
> The DataNode creates a BPServiceActor for \{{mycluster:8020}} that can never
> connect. The DataNode log fills with:
> {noformat}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to
> server: mycluster:8020
> {noformat}
> After running for some time the DataNode heap is dominated by
> {\{ReceivedDeletedBlockInfo}} / \{{Block}} / \{{HashMap$Node}} entries
> belonging to
> that dead actor's IBR queue. A production heap dump showed ~223M
> {\{ReceivedDeletedBlockInfo}} instances occupying ~30 GB.
> h2. Impact
> * DataNode OOM / crash.
> * After a restart the heap starts filling again immediately, because the
> configuration is unchanged.
> * Affects any cluster where a NameNode is permanently removed/decommissioned
> without updating \{{dfs.ha.namenodes.<nsId>}}, or where an RPC address is
> simply misconfigured.
> h2. Proposed fix
> # Add a configurable cap on the pending IBR queue,
> {\{dfs.datanode.ibr.max.pending.size}} (default 1,000,000). When exceeded, the
> queue is cleared and a warning is logged.
> # Add a staleness guard \{{dfs.datanode.ibr.max.stale.interval.ms}}
> (default 30 min): if there has been no successful IBR send within this window
> while entries are pending, the queue is cleared.
> # After clearing (overflow or staleness), schedule a Full Block Report so the
> NameNode gets a complete, consistent view once it is reachable again.
> This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an
> optimization for incremental updates between FBRs. Dropping queued IBRs for an
> unreachable NN cannot cause inconsistency because a fresh FBR is sent on
> reconnect/re-register.
> h2. Test plan
> New unit tests in \{{TestIncrementalBlockReportManager}}:
> * \{{testIBRQueueSizeLimit}} — queue is capped and cleared on overflow.
> * \{{testIBRQueueStaleInterval}} — queue is cleared after the stale interval.
> * \{{testIBRQueueNoLimit}} — cap disabled (0) keeps old behavior.
> * \{{testIBRDeduplication}} — per-block dedup still works.
> * \{{testTotalPendingIBRSizeMultipleStorages}} — size counting across
> storages.
> ---
> ## Labels (可选)
> datanode, ibr, memory-leak, oom
> ## 关联关系
> - Relates to / Is related to: **HDFS-9917**, **HDFS-15113**
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
