[
https://issues.apache.org/jira/browse/HDFS-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18071978#comment-18071978
]
ASF GitHub Bot commented on HDFS-16942:
---------------------------------------
cxzl25 commented on code in PR #5460:
URL: https://github.com/apache/hadoop/pull/5460#discussion_r3051817271
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##########
@@ -791,6 +792,9 @@ private void offerService() throws Exception {
shouldServiceRun = false;
return;
}
+ if (InvalidBlockReportLeaseException.class.getName().equals(reClass)) {
+ fullBlockReportLeaseId = 0;
Review Comment:
## Description
HDFS-16942 introduced `InvalidBlockReportLeaseException`, which the NameNode
now throws back to the DataNode via RPC when a block report is rejected due to
an invalid lease. On a DataNode that also includes HDFS-16942, the exception is
caught and `fullBlockReportLeaseId` is reset to 0, allowing the DN to request a
new lease on the next heartbeat and retry.
However, during a rolling upgrade where the NameNode has been upgraded (with
HDFS-16942) but DataNodes are still running an older version (without
HDFS-16942), the old DataNode code does not have the
`InvalidBlockReportLeaseException` handling branch in
`BPServiceActor.offerService()`. This causes the DN to enter an infinite
failure loop where it can never successfully send a full block report.
## Root Cause
In `BPServiceActor.offerService()`, the logic works as follows:
1. The DN requests a block report lease during heartbeat only when
`fullBlockReportLeaseId == 0`:
```java
boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
scheduler.isBlockReportDue(startTime);
```
2. After sending the block report, `fullBlockReportLeaseId` is reset to 0:
```java
if ((fullBlockReportLeaseId != 0) || forceFullBr) {
cmds = blockReport(fullBlockReportLeaseId);
fullBlockReportLeaseId = 0; // not reached if blockReport() throws
}
```
3. When the upgraded NN throws `InvalidBlockReportLeaseException`,
`blockReport()` propagates the exception. The `fullBlockReportLeaseId = 0` line
after the call is never executed.
4. The exception is caught by the generic `RemoteException` catch block. The
old DN code does not recognize `InvalidBlockReportLeaseException`, so
`fullBlockReportLeaseId` remains set to the stale invalid value.
5. On the next heartbeat iteration, because `fullBlockReportLeaseId != 0`,
`requestBlockReportLease` is `false` — the DN does not request a new lease. It
then attempts `blockReport()` again with the same stale lease, which the NN
rejects again. This repeats indefinitely.
> Send error to datanode if FBR is rejected due to bad lease
> ----------------------------------------------------------
>
> Key: HDFS-16942
> URL: https://issues.apache.org/jira/browse/HDFS-16942
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
> Affects Versions: 3.4.0, 3.3.6
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.5, 3.3.6
>
>
> When a datanode sends a FBR to the namenode, it requires a lease to send it.
> On a couple of busy clusters, we have seen an issue where the DN is somehow
> delayed in sending the FBR after requesting the least. Then the NN rejects
> the FBR and logs a message to that effect, but from the Datanodes point of
> view, it thinks the report was successful and does not try to send another
> report until the 6 hour default interval has passed.
> If this happens to a few DNs, there can be missing and under replicated
> blocks, further adding to the cluster load. Even worse, I have see the DNs
> join the cluster with zero blocks, so it is not obvious the under replication
> is caused by lost a FBR, as all DNs appear to be up and running.
> I believe we should propagate an error back to the DN if the FBR is rejected,
> that way, the DN can request a new lease and try again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
