emanuilov commented on code in PR #3605: URL: https://github.com/apache/iceberg-rust/pull/3605#discussion_r4445465446
########## crates/iceberg/src/io/storage/config/s3.rs: ########## @@ -65,6 +65,13 @@ pub const S3_ALLOW_ANONYMOUS: &str = "s3.allow-anonymous"; pub const S3_DISABLE_EC2_METADATA: &str = "s3.disable-ec2-metadata"; /// Option to skip loading configuration from config file and the env. pub const S3_DISABLE_CONFIG_LOAD: &str = "s3.disable-config-load"; +/// Number of times to retry an S3 operation, `0` to disable retrying. Retries back off exponentially, +/// with jitter, between [`S3_RETRY_MIN_WAIT_MS`] and [`S3_RETRY_MAX_WAIT_MS`]. +pub const S3_RETRY_NUM_RETRIES: &str = "s3.retry.num-retries"; +/// Minimum wait in milliseconds before retrying an S3 operation. +pub const S3_RETRY_MIN_WAIT_MS: &str = "s3.retry.min-wait-ms"; +/// Maximum wait in milliseconds between two retries of an S3 operation. +pub const S3_RETRY_MAX_WAIT_MS: &str = "s3.retry.max-wait-ms"; Review Comment: Just to check I understand: do you mean a generic set of retry keys in the opendal namespace that every OpenDAL backend uses, with s3.retry.* taking precedence for S3? If so, here’s what I’d do: - Add opendal.retry.num-retries, opendal.retry.min-wait-ms and opendal.retry.max-wait-ms to OpenDalClientConfig, next to opendal.io-timeout-ms, and build every backend’s RetryLayer from them. - For S3, any s3.retry.* key that is set overrides the matching field, so the order is s3.retry.*, then opendal.retry.*, then OpenDAL’s defaults. Each key is resolved separately, so you can set one at either layer. - Java-configured tables work unchanged. Anyone who wants the same retries on every backend sets the opendal.retry.* keys once. - Tests for generic-only, S3-only, both set (S3 wins), a key set at only one layer, and a non-S3 backend ignoring s3.retry.*. Is that what you had in mind? If so, I’ll add it to this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
