steveloughran commented on a change in pull request #1838: HADOOP-16711 Add way
to skip verifyBuckets check in S3A fs init()
URL: https://github.com/apache/hadoop/pull/1838#discussion_r377552989
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -452,6 +450,33 @@ public void initialize(URI name, Configuration
originalConf)
}
+ /**
+ * Test bucket existence in S3.
+ * When value of {@link Constants#S3A_BUCKET_PROBE is set to 0 by client,
+ * bucket existence check is not done to improve performance of
+ * S3AFileSystem initialisation. When set to 1 or 2, bucket existence check
+ * will be performed which is potentially slow.
+ * @throws IOException
+ */
+ private void doBucketProbing() throws IOException {
Review comment:
you just need to add the @RetryPolicy on the method based on the inner ones.
it's not about actually doing the retries, just declare what it is for people
looking at it.
The goal is that if we can keep those attributes accurate you just need to
look at the method and determine the complete retrial policy that it has -all
the way down.
This means we need to add the attribute to all methods, and keep an eye on
them to make sure that they don't go invalid/out of date after changes
underneath.
It's a shame we can't automate this -but the need to have them does force us
to audit the code. Like you say: we mustn't retry around a retry -but we must
have a retry somewhere above every non-retried operation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
