singer-bin commented on PR #8626: URL: https://github.com/apache/hadoop/pull/8626#issuecomment-5067448567
Thanks for the review, @Hexiaoqiao ! You are right — the pending-block-count cap alone is enough to bound the DataNode heap footprint, so `maxStaleIntervalMs` was redundant. I have removed it and the related config (`dfs.datanode.ibr.max.stale.interval.ms`). The fix now keeps a single, simple guard: `dfs.datanode.ibr.max.pending.blocks` (default 1,000,000). The pending block count is tracked in O(1), and when it reaches the cap `IncrementalBlockReportManager#clearIBRsIfNeeded()` clears the queue and the `BPServiceActor` schedules a full block report so the NameNode can resync once it is reachable again. Set the value to 0 to disable. Updated tests (all passing locally): - `TestIncrementalBlockReportManager`: counter accounting, dedup, size-cap clear, and cap-disabled behavior. - `TestIncrementalBlockReports#testIBRQueueBoundedWhenNNUnreachable`: end-to-end test with a MiniDFSCluster and an unreachable NN, asserting the pending block count never exceeds the cap. Please take another look. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
