emanuilov opened a new issue, #3604: URL: https://github.com/apache/iceberg-rust/issues/3604
### Apache Iceberg Rust version 0.10.1 ### Describe the bug ### Is your feature request related to a problem or challenge? We run a high-throughput writer that streams Kafka topics into Amazon S3 Tables using `iceberg`, `iceberg-catalog-s3tables` and `iceberg-storage-opendal`. When many data files are uploaded at once (for example a backfill flushing hundreds of partitions, or dozens of writers creating files in new tables in one table bucket), S3 answers `503 SlowDown` while it scales the prefix up. That is expected S3 behaviour, and AWS's guidance is to retry with exponential backoff and jitter. The opendal S3 operator is always wrapped in `RetryLayer::new()`: 3 retries starting at 1 s, factor 2, no jitter. There is no property or builder option to change it (the per-operation timeout became configurable in #3263, the retry policy did not). So: - the error reaches the caller after about 7 s, while S3's scaling usually needs longer, so a whole commit cycle fails and has to be redone, which adds to the load that caused the throttling; - without jitter, all uploads throttled in the same burst retry at the same moments and get throttled again together. The only workaround today is a custom `StorageFactory` that re-implements the opendal storage just to change one layer. ### Describe the solution you'd like Read Iceberg Java's existing S3FileIO properties, with the same meaning, so one catalog configuration behaves the same from Java and Rust: - `s3.retry.num-retries` - `s3.retry.min-wait-ms` - `s3.retry.max-wait-ms` Define the keys once in `iceberg::io` (next to the other `S3_*` constants), so any S3 backend can read them: the opendal backend now, and the `object_store` backend proposed in #3165 later (it maps naturally onto `object_store`'s `RetryConfig`). For the opendal backend: map them onto the S3 operator's `RetryLayer` (`with_max_times`, `with_min_delay`, `with_max_delay`) and enable jitter whenever any of them is set. Java has no jitter key, and the AWS SDK always jitters. With none set, the layer stays `RetryLayer::new()`, so existing users see no change. Other backends are not affected. ### Willingness to contribute I've prepared a PR implementing this for the opendal backend (unit tests included) and will link it here. ## AI Notice Drafted with AI ### To Reproduce _No response_ ### Expected behavior _No response_ ### Willingness to contribute _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
