SandeepSinghGahir commented on issue #10340: URL: https://github.com/apache/iceberg/issues/10340#issuecomment-2353801275
> I can't speak for the S3FileIO developers; S3AFS is where I code and while there's a lot of work there for recovery [here](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java) and elsewhere, we are all still finding obscure recovery failures one by one, such as how the AWS SDK doesn't recovery properly [if a multipart part upload fails with a 500](https://github.com/apache/hadoop/pull/6938). > > 1. If you want to use the S3FileIO: try those options. > > 2. If you want an S3 client which has fixes for all the failures we've hit: S3A is your friend. > > 3. Or you take up the PR, do your own iceberg release with it and let everyone know if it does/doesn't work. Real world pre-release testing is the way to do this I tried retry options with S3FileIO but I don't see any improvement. Some days the job succeeds without issues some days it needs 1 retry and some days 5. So, no config seems to work here. I have also tried your suggestions in previous comment: using Hadoop s3a or increase values for `aws.retryMode` and `aws.maxAttempts`, but that also didn't help. I can try with a custom S3A client. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
