amogh-jahagirdar commented on code in PR #11052:
URL: https://github.com/apache/iceberg/pull/11052#discussion_r1764104918
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java:
##########
@@ -393,6 +403,21 @@ public class S3FileIOProperties implements Serializable {
*/
private static final String S3_FILE_IO_USER_AGENT = "s3fileio/" +
EnvironmentContext.get();
+ /** Number of times to retry S3 operations. */
+ public static final String S3_RETRY_NUM_RETRIES = "s3.retry.num-retries";
+
+ public static final int S3_RETRY_NUM_RETRIES_DEFAULT = 32;
Review Comment:
@ookumuso This value seems really high to me? This can compound considering
engine integrations may already have their own level of retries on top of this
(e.g. Spark task retries). I would also hypothesize that the chance of success
after some small number of retries (e.g. I think the SDK default is 5?) is
slim and that it's generally not worth trying beyond that.
In the performed load tests did this 32 number markedly improved reliability
generally compared to lower values?
I think setting this default value comes down to what is the typical
distribution of outcomes and I'd speculate that in the average case when there
is 5xx errors this large number of retries won't help? In some extreme cases it
seems possible, but I think for those cases, users can override the config
defined here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
