noslowerdna commented on a change in pull request #624: HADOOP-15999. S3Guard:
Better support for out-of-band operations
URL: https://github.com/apache/hadoop/pull/624#discussion_r269690083
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2382,6 +2392,38 @@ S3AFileStatus innerGetFileStatus(final Path f,
"deleted by S3Guard");
}
+ // if ms is not authoritative, check S3 if there's any recent
+ // modification - compare the modTime to check if metadata is up to date
+ // Skip going to s3 if the file checked is a directory. Because if the
+ // dest is also a directory, there's no difference.
+ // TODO After HADOOP-16085 the modification detection can be done with
+ // etags or object version instead of modTime
+ if (!pm.getFileStatus().isDirectory() &&
+ !allowAuthoritative) {
+ LOG.debug("Metadata for {} found in the non-auth metastore.", path);
+ final long msModTime = pm.getFileStatus().getModificationTime();
+
+ S3AFileStatus s3AFileStatus;
+ try {
+ s3AFileStatus = s3GetFileStatus(path, key, tombstones);
+ } catch (FileNotFoundException fne) {
+ s3AFileStatus = null;
+ }
+ if (s3AFileStatus == null) {
+ LOG.warn("Failed to find file {}. Either it is not yet visible, or "
+ + "it has been deleted.", path);
+ } else {
+ final long s3ModTime = s3AFileStatus.getModificationTime();
+
+ if(s3ModTime > msModTime) {
+ LOG.debug("S3Guard metadata for {} is outdated, updating it",
Review comment:
might include the 2 mod times in the debug log
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
