[
https://issues.apache.org/jira/browse/HDFS-17722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062423#comment-18062423
]
ASF GitHub Bot commented on HDFS-17722:
---------------------------------------
balodesecurity opened a new pull request, #8295:
URL: https://github.com/apache/hadoop/pull/8295
## Problem
On a standby NameNode, a DataNode can get stuck in the
`DECOMMISSION_INPROGRESS` state indefinitely when a timing race causes a
replica to be flagged as **excess** instead of **live** during decommissioning.
Sequence:
1. File is written to DN-A, DN-B, DN-C (RF=3).
2. DN-A is marked for decommission.
3. The block manager schedules re-replication → copies a new replica to DN-D.
4. On the standby NN, the block report for DN-D arrives *before* the
decommission state for DN-A is propagated. The standby marks DN-D's replica as
**excess** (it looks like an over-replicated block).
5. The decommission monitor on the standby calls `isSufficient()`:
`numLive=2` (DN-B, DN-C) satisfies RF=3? No. It sees only 2 live copies, so
decommission stalls.
6. Meanwhile DN-A is never fully decommissioned because `isSufficient()`
never returns true.
The excess replica on DN-D is a **physically present block copy** and
contributes to durability — ignoring it causes the deadlock.
## Fix
In `DatanodeAdminManager.isSufficient()`, count excess replicas alongside
live replicas for the sufficiency check on non-under-construction blocks:
```java
final int numLiveAndExcess = numLive + numberReplicas.excessReplicas();
if (numLiveAndExcess >= blockManager.getDefaultStorageNum(block)
&& blockManager.hasMinStorage(block, numLive)) {
return true;
}
```
The `hasMinStorage` guard (checks `dfs.replication.min`, default 1) ensures
decommission does not proceed if zero live replicas exist — excess-only
replicas are not guaranteed durable. After decommission completes, if the
excess replica on DN-D is subsequently deleted, the block manager's normal
under-replication detection will schedule re-replication.
## Testing
**Unit tests** — `TestDatanodeAdminManagerIsSufficient` (5 tests, no cluster
required):
| Test | Scenario | Expected |
|---|---|---|
| `testExcessReplicaCountsTowardSufficiency` | HDFS-17722 bug: live=1,
excess=1, RF=2 | `true` |
| `testNormalDecommissionStillSufficient` | Baseline: live=2, excess=0, RF=2
| `true` |
| `testNoLiveReplicaBlocksDecommission` | Safety guard: live=0, excess=2,
RF=2 | `false` |
| `testInsufficientEvenWithExcess` | live=0, excess=1, RF=2 — not enough
either way | `false` |
| `testExcessAboveRFWithMinLive` | live=1, excess=2, RF=2 — excess
over-covers RF | `true` |
```
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
```
**Docker integration** — 3-DataNode cluster with 1 NameNode and RF=3, 5
scenarios:
- Scenario 1: Clean decommission (RF=2) — **PASS**
- Scenario 2: RF=3→2 creates excess replicas, then decommission DN2 —
**PASS**
- Scenario 3: Same scenario on DN3 — **PASS**
- Scenario 4: Repeated decommission + recommission cycles (3 rounds) —
**PASS**
- Scenario 5: Data integrity check after decommission — **PASS**
```
Results: 0 failure(s) — ALL TESTS PASSED
```
## Related
- JIRA: https://issues.apache.org/jira/browse/HDFS-17722
> DataNode stuck in decommissioning on standby NameNode
> -----------------------------------------------------
>
> Key: HDFS-17722
> URL: https://issues.apache.org/jira/browse/HDFS-17722
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.3.6
> Reporter: Benoit Sigoure
> Priority: Minor
>
> When decommissioning a DataNode in our cluster, we observed a situation where
> the active NameNode had marked the DataNode as decommissioned but the standby
> had it stuck in decommissioning state indefinitely (we waited 8h) due to a
> block being allegedly under replicated (note: for this path the target
> replication factor is 2x). The standby NameNode kept logging this in a loop:
> {{2025-01-31 12:02:35,963 INFO BlockStateChange: Block:
> blk_1486338012_426727507, Expected Replicas: 2, live replicas: 1, corrupt
> replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1,
> maintenance replicas: 0, live entering maintenance replicas: 0, replicas on
> stale nodes: 0, readonly replicas: 0, excess replicas: 1, Is Open File:
> false, Datanodes having this block: 10.128.89.32:9866 10.128.118.216:9866
> 10.128.49.6:9866 , Current Datanode: 10.128.118.216:9866, Is current datanode
> decommissioning: true, Is current datanode entering maintenance: false}}
> Looking at the fsck report for this block, the active NameNode was reporting
> the following:
> {code:java}
> Block Id: blk_1486338012
> Block belongs to: /path/to/file
> No. of Expected Replica: 2
> No. of live Replica: 2
> No. of excess Replica: 0
> No. of stale Replica: 0
> No. of decommissioned Replica: 1
> No. of decommissioning Replica: 0
> No. of corrupted Replica: 0
> Block replica on datanode/rack: datanode-v3-25-hadoop.hadoop/default-rack is
> HEALTHY
> Block replica on datanode/rack: datanode-v3-39-hadoop.hadoop/default-rack is
> DECOMMISSIONED
> Block replica on datanode/rack: datanode-v3-26-hadoop.hadoop/default-rack is
> HEALTHY
> {code}
> Whereas on the standby it says:
> {code:java}
> Block Id: blk_1486338012
> Block belongs to: /path/to/file
> No. of Expected Replica: 2
> No. of live Replica: 1
> No. of excess Replica: 1
> No. of stale Replica: 0
> No. of decommissioned Replica: 0
> No. of decommissioning Replica: 1
> No. of corrupted Replica: 0
> Block replica on datanode/rack: datanode-v3-25-hadoop.hadoop/default-rack is
> HEALTHY
> Block replica on datanode/rack: datanode-v3-39-hadoop.hadoop/default-rack is
> DECOMMISSIONING
> Block replica on datanode/rack: datanode-v3-26-hadoop.hadoop/default-rack is
> HEALTHY
> {code}
> {code:java}
> hadoop@namenode-0:/$ hdfs dfs -ls /path/to/file
> -rw-r--r-- 2 hbase supergroup 32453388896 2025-01-02 16:15 /path/to/file
> {code}
> After restarting the standby NameNode, the problem disappeared, the datanode
> in question transitioned to decommissioned state as expected.
> Credits for the bug report go to Tomas Baltrunas at Arista.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
