[
https://issues.apache.org/jira/browse/HDFS-17906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18072526#comment-18072526
]
ASF GitHub Bot commented on HDFS-17906:
---------------------------------------
cxzl25 opened a new pull request, #8416:
URL: https://github.com/apache/hadoop/pull/8416
### Description of PR
## Description
HDFS-16942 introduced `InvalidBlockReportLeaseException`, which the NameNode
now throws back to the DataNode via RPC when a block report is rejected due to
an invalid lease. On a DataNode that also includes HDFS-16942, the exception is
caught and `fullBlockReportLeaseId` is reset to 0, allowing the DN to request a
new lease on the next heartbeat and retry.
However, during a rolling upgrade where the NameNode has been upgraded (with
HDFS-16942) but DataNodes are still running an older version (without
HDFS-16942), the old DataNode code does not have the
`InvalidBlockReportLeaseException` handling branch in
`BPServiceActor.offerService()`. This causes the DN to enter an infinite
failure loop where it can never successfully send a full block report.
## Root Cause
In `BPServiceActor.offerService()`, the logic works as follows:
1. The DN requests a block report lease during heartbeat only when
`fullBlockReportLeaseId == 0`:
```java
boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
scheduler.isBlockReportDue(startTime);
```
2. After sending the block report, `fullBlockReportLeaseId` is reset to 0:
```java
if ((fullBlockReportLeaseId != 0) || forceFullBr) {
cmds = blockReport(fullBlockReportLeaseId);
fullBlockReportLeaseId = 0; // not reached if blockReport() throws
}
```
3. When the upgraded NN throws `InvalidBlockReportLeaseException`,
`blockReport()` propagates the exception. The `fullBlockReportLeaseId = 0` line
after the call is never executed.
4. The exception is caught by the generic `RemoteException` catch block. The
old DN code does not recognize `InvalidBlockReportLeaseException`, so
`fullBlockReportLeaseId` remains set to the stale invalid value.
5. On the next heartbeat iteration, because `fullBlockReportLeaseId != 0`,
`requestBlockReportLease` is `false` — the DN does not request a new lease. It
then attempts `blockReport()` again with the same stale lease, which the NN
rejects again. This repeats indefinitely.
### How was this patch tested?
Add test
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> Rolling upgrade: old DataNodes get stuck in infinite invalid block report
> lease loop
> ------------------------------------------------------------------------------------
>
> Key: HDFS-17906
> URL: https://issues.apache.org/jira/browse/HDFS-17906
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: dzcxzl
> Priority: Major
>
>
> HDFS-16942 introduced `InvalidBlockReportLeaseException`, which the NameNode
> now throws back to the DataNode via RPC when a block report is rejected due
> to an invalid lease. On a DataNode that also includes HDFS-16942, the
> exception is caught and `fullBlockReportLeaseId` is reset to 0, allowing the
> DN to request a new lease on the next heartbeat and retry.
> However, during a rolling upgrade where the NameNode has been upgraded (with
> HDFS-16942) but DataNodes are still running an older version (without
> HDFS-16942), the old DataNode code does not have the
> `InvalidBlockReportLeaseException` handling branch in
> `BPServiceActor.offerService()`. This causes the DN to enter an infinite
> failure loop where it can never successfully send a full block report.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
