arifazmidd opened a new issue, #11412: URL: https://github.com/apache/iceberg/issues/11412
### Apache Iceberg version 1.6.1 (latest release) ### Query engine Spark ### Please describe the bug 🐞 There seems to be some transient AWS connection issues with running Spark. I have workloads running Spark on both DBR and EMR which are interacting with Iceberg tables. They run fine most of the time but randomly will fail with the following error: ``` software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain ``` On DBR it is randomly failing with the [overwritePartitions](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameWriterV2.overwritePartitions.html#pyspark.sql.DataFrameWriterV2.overwritePartitions) call from the DataFrameWriterV2 and on EMR it is randomly failing with the [spark sql procedure for expiring snapshots](https://iceberg.apache.org/docs/1.5.1/spark-procedures/#expire_snapshots) I believe the issue could be similar to this: https://github.com/aws/aws-sdk-js-v3/issues/4867. Basically that we need to be mindful of how many requests we send to IMDS to avoid throttling errors. I'm thinking that with Iceberg we are creating a new client for every call which could be causing this? I will provide my config details and the full error log as comments. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [X] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
