danielcweeks commented on code in PR #11052:
URL: https://github.com/apache/iceberg/pull/11052#discussion_r1777369877
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java:
##########
@@ -824,6 +889,65 @@ public <T extends S3ClientBuilder> void
applyEndpointConfigurations(T builder) {
}
}
+ /**
+ * Override the retry configurations for an S3 client.
+ *
+ * <p>Sample usage:
+ *
+ * <pre>
+ *
S3Client.builder().applyMutation(s3FileIOProperties::applyRetryConfigurations)
+ * </pre>
+ */
+ public <T extends S3ClientBuilder> void applyRetryConfigurations(T builder) {
+ builder.overrideConfiguration(
+ config ->
+ config.retryPolicy(
+ // Use a retry strategy which will persistently retry
throttled exceptions with
+ // exponential backoff, to give S3 a chance to autoscale.
+ // LEGACY mode works best here, as it will allow throttled
exceptions to use all of
+ // the configured retry attempts.
+ RetryPolicy.builder(RetryMode.LEGACY)
+ .numRetries(s3RetryNumRetries)
+ .throttlingBackoffStrategy(
+ EqualJitterBackoffStrategy.builder()
+ .baseDelay(Duration.ofMillis(s3RetryMinWaitMs))
+
.maxBackoffTime(Duration.ofMillis(s3RetryMaxWaitMs))
+ .build())
+
+ // Workaround: add XMLStreamException as a retryable
exception.
+ // https://github.com/aws/aws-sdk-java-v2/issues/5442
+ // Without this workaround, we see SDK failures if there's
a socket exception
+ // while parsing an error XML response.
+ .retryCondition(
+ OrRetryCondition.create(
+ RetryCondition.defaultRetryCondition(),
+
RetryOnExceptionsCondition.create(XMLStreamException.class)))
+
+ // Workaround: exclude all 503s from consuming retry
tokens.
Review Comment:
Based on this it sounds like you're proposing unlimited retries on 503s and
32 retries on other failures? This feels far too aggressive, but I'm not sure
if I'm reading this correctly. Why would we treat bucket HEAD requests
differently anyway? (Also, I'm not sure that we actually make that request
currently, so I don't think this is necessary)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
