ajfabbri commented on a change in pull request #951: HADOOP-15183. S3Guard
store becomes inconsistent after partial failure of rename
URL: https://github.com/apache/hadoop/pull/951#discussion_r293168151
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
##########
@@ -461,22 +509,26 @@ public static void addMoveAncestors(MetadataStore ms,
}
}
- public static void addAncestors(MetadataStore metadataStore,
- Path qualifiedPath, String username) throws IOException {
- Collection<PathMetadata> newDirs = new ArrayList<>();
- Path parent = qualifiedPath.getParent();
- while (!parent.isRoot()) {
- PathMetadata directory = metadataStore.get(parent);
- if (directory == null || directory.isDeleted()) {
- S3AFileStatus s3aStatus = new S3AFileStatus(Tristate.FALSE, parent,
username);
- PathMetadata meta = new PathMetadata(s3aStatus, Tristate.FALSE, false);
- newDirs.add(meta);
- } else {
- break;
- }
- parent = parent.getParent();
- }
- metadataStore.put(newDirs);
+ /**
+ * This adds all new ancestors of a path as directories.
+ * This forwards to
+ * {@link MetadataStore#addAncestors(Path, BulkOperationState)}.
+ * <p>
+ * Originally it implemented the logic to probe for an add ancestors,
+ * but with the addition of a store-specific bulk operation state
+ * it became unworkable.
+ *
Review comment:
Wouldn't hold up this patch for it, but am curious how this became
unworkable. I imagine de-duping metadata writes (ancestors) using a bulk
context, such that S3A repeating ancestor writes within an op are not an issue.
Again, always trying to keep MetadataStore as simple as possible and
specific to logging metadata operations on a FileSystem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
