steveloughran commented on a change in pull request #2149:
URL: https://github.com/apache/hadoop/pull/2149#discussion_r467963869
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2890,72 +2942,114 @@ S3AFileStatus innerGetFileStatus(final Path f,
// Skip going to s3 if the file checked is a directory. Because if the
// dest is also a directory, there's no difference.
- if (!pm.getFileStatus().isDirectory() &&
+ if (!msStatus.isDirectory() &&
!allowAuthoritative &&
probes.contains(StatusProbeEnum.Head)) {
// a file has been found in a non-auth path and the caller has not said
// they only care about directories
LOG.debug("Metadata for {} found in the non-auth metastore.", path);
- final long msModTime = pm.getFileStatus().getModificationTime();
-
- S3AFileStatus s3AFileStatus;
- try {
- s3AFileStatus = s3GetFileStatus(path, key, probes, tombstones);
- } catch (FileNotFoundException fne) {
- s3AFileStatus = null;
- }
- if (s3AFileStatus == null) {
- LOG.warn("Failed to find file {}. Either it is not yet visible, or "
- + "it has been deleted.", path);
- } else {
- final long s3ModTime = s3AFileStatus.getModificationTime();
-
- if(s3ModTime > msModTime) {
- LOG.debug("S3Guard metadata for {} is outdated;"
- + " s3modtime={}; msModTime={} updating metastore",
- path, s3ModTime, msModTime);
- return S3Guard.putAndReturn(metadataStore, s3AFileStatus,
- ttlTimeProvider);
+ // If the timestamp of the pm is close to "now", we don't need to
Review comment:
I should add that just as #2168 pulls out the openfile logic into its
own class, I did think about doing that for getFileStatus...but given its
critical to backport the core probes, decided against it. Maybe later
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
