[
https://issues.apache.org/jira/browse/HADOOP-18242?focusedWorklogId=784630&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784630
]
ASF GitHub Bot logged work on HADOOP-18242:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jun/22 15:20
Start Date: 24/Jun/22 15:20
Worklog Time Spent: 10m
Work Description: steveloughran commented on code in PR #4331:
URL: https://github.com/apache/hadoop/pull/4331#discussion_r906166340
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -531,13 +541,40 @@ public Pair<AbfsRestOperation, Boolean> renamePath(
url,
requestHeaders);
try {
+ incrementAbfsRenamePath();
op.execute(tracingContext);
- return Pair.of(op, false);
+ // AbfsClientResult contains the AbfsOperation, If recovery happened or
+ // not, and the incompleteMetaDataState is true or false.
+ // If we successfully rename a path and isMetadataIncompleteState was
+ // true, then rename was recovered, else it didn't, this is why
+ // isMetadataIncompleteState is used for renameRecovery(as the 2nd
param).
+ return new AbfsClientRenameResult(op, isMetadataIncompleteState,
isMetadataIncompleteState);
} catch (AzureBlobFileSystemException e) {
// If we have no HTTP response, throw the original exception.
if (!op.hasResult()) {
throw e;
}
+
+ // ref: HADOOP-18242. Rename failure occurring due to a rare case of
+ // tracking metadata being in incomplete state.
+ if (op.getResult().getStorageErrorCode()
+ .equals(RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode())
+ && !isMetadataIncompleteState) {
+ //Logging
+ ABFS_METADATA_INCOMPLETE_RENAME_FAILURE
+ .info("Rename Failure attempting to resolve tracking metadata
state and retrying.");
+
+ // Doing a HEAD call resolves the incomplete metadata state and
+ // then we can retry the rename operation.
+ getPathStatus(source, false, tracingContext);
Review Comment:
I've had one more thought here. the path status contains the etag, doesn't
it? so if sourceEtag was null, now we can set it. That way if the rename
failure is followed
immediately buy the rename-failure-but-it-really-happened event of
HADOOP-18163, we are lined up for recovery
Issue Time Tracking
-------------------
Worklog Id: (was: 784630)
Time Spent: 4h 20m (was: 4h 10m)
> ABFS Rename Failure when tracking metadata is in incomplete state
> -----------------------------------------------------------------
>
> Key: HADOOP-18242
> URL: https://issues.apache.org/jira/browse/HADOOP-18242
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> If a node in the datacenter crashes while processing an operation,
> occasionally it can leave the Storage-internal blob tracking metadata in an
> incomplete state. We expect this to happen occasionally, and so all API’s
> are designed in such a way that if this incomplete state is observed on a
> blob, the situation is resolved before the current operation proceeds.
> However, this incident has exposed a bug specifically with the Rename API,
> where the incomplete state fails to resolve, leading to this incorrect
> failure. As a temporary mitigation, if any other operation is performed on
> this blob – GetBlobProperties, GetBlob, GetFileProperties, SetFileProperties,
> etc – it should resolve the incomplete state, and rename will no longer hit
> this issue.
> StackTrace:
> {code:java}
> 2022-03-22 17:52:19,789 DEBUG [regionserver/euwukwlss-hg50:16020.logRoller]
> services.AbfsClient: HttpRequest:
> 404,RenameDestinationParentPathNotFound,cid=ef5cbf0f-5d4a-4630-8a59-3d559077fc24,rid=35fef164-101f-000b-1b15-3ed818000000,sent=0,recv=212,PUT,https://euwqdaotdfdls03.dfs.core.windows.net/eykbssc/apps/hbase/data/oldWALs/euwukwlss-hg50.tdf.qa%252C16020%252C1647949929877.1647967939315?timeout=90
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
