[
https://issues.apache.org/jira/browse/HDFS-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063951#comment-18063951
]
ASF GitHub Bot commented on HDFS-17840:
---------------------------------------
balodesecurity opened a new pull request, #8326:
URL: https://github.com/apache/hadoop/pull/8326
## Summary
The **Nodes in Service** count displayed per storage type in the NameNode UI
(DFS Storage Types section) could become grossly incorrect — e.g. showing 3
nodes when the cluster has 26.
### Root Cause
`DatanodeStats.StorageTypeStatsMap` maintains a `StorageTypeStats` entry per
storage type with an incremental `nodesInService` counter. The map entry was
removed whenever `nodesInService` dropped to 0 — even when
decommissioning/maintenance nodes still used the same storage type.
The premature removal caused a cascade:
1. Node A (last in-service node for DISK) starts decommissioning →
`nodesInService` drops to 0 → **DISK entry removed**.
2. Next heartbeat from any node recreates the entry fresh (`nodesInService =
0`).
3. When in-service node B heartbeats: `subtract(B)` runs against the fresh
entry → `nodesInService: 0→-1`. Then `add(B)` → `nodesInService: -1→0`. **B's
in-service contribution is lost**.
4. After enough such cycles the reported count is far below the real number.
### Fix
Add a `totalNodes` counter to `StorageTypeStats` that tracks **all** nodes
using a storage type (in-service + decommissioning + maintenance). Change the
map-entry removal condition from `nodesInService == 0` to `totalNodes == 0`. An
entry is now only removed when no node of any admin state still uses that
storage type.
**Changed files:**
- `StorageTypeStats.java` — new `totalNodes` field; `addNode`/`subtractNode`
always update it; new `getTotalNodes()` accessor
- `DatanodeStats.java` — removal condition updated to `getTotalNodes() == 0`
- `TestStorageTypeStatsMap.java` — 4 new unit tests (new file)
## Test plan
- [x] `TestStorageTypeStatsMap` (4 tests) — PASS
- `testBasicAddRemove` — basic correctness
- `testEntryNotRemovedWhenDecommissioningNodeRemains` — entry survives
when a decommissioning node still uses the storage type; nodesInService stays
correct
- `testEntryNotRemovedWhenLastInServiceDecommissions` — entry survives
when the last in-service node decommissions; new in-service node is counted
correctly
- `testEntryRemovedOnlyWhenAllNodesGone` — entry removed only after all
nodes (including decommissioning) are gone
- [ ] Full blockmanagement test suite (CI)
> The "Nodes in Service" information is incorrect under DFS Storage Types in
> NameNode UI
> --------------------------------------------------------------------------------------
>
> Key: HDFS-17840
> URL: https://issues.apache.org/jira/browse/HDFS-17840
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.2.2
> Reporter: terrytlu
> Priority: Major
> Attachments: before_cdp_upgrade_namenode.png,
> image-2025-09-23-11-34-26-420.png, latest_nodes_in_service_namenode.png
>
>
> We found in our hadoop cluster, which has 26 nodes, while in NameNode UI,
> only show 3 nodes
> !image-2025-09-23-11-34-26-420.png|width=720,height=253!
> the problem still exists after merge
> https://issues.apache.org/jira/browse/HDFS-16402 ,this metric modification
> involves multiple data fields and heartbeat logic, it is difficult to
> guarantee atomicity.
>
> And the metrics are calculated using real-time increments and decrements,
> once it is corrupted, it cannot be restored. We recommend change the
> implementation to retrieve the data from
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager#datanodeMap to
> ensure that data is neither lost nor duplicated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
