MonkeyCanCode commented on issue #11843: URL: https://github.com/apache/iceberg/issues/11843#issuecomment-2568787480
@rohitanil as commented by @stym06 , you are missing couple things and one of them is aws jar. If u want to load hadoop-aws jars, you will need to load couple additional dependencies jars as well. Alternative, you can use iceberg-aws-bundle jar instead. This is documented in https://iceberg.apache.org/docs/1.7.1/aws/#spark Other than this, you will need couple additional properties so Spark will know how to talk to S3, such as following: ``` spark.hadoop.fs.s3.impl org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem # If u are planning to pass in key and secret spark.hadoop.fs.s3a.access.key=ACCESSKEY spark.hadoop.fs.s3a.secret.key=SECRETKEY ## if u are planning to use assume role, then set the next one and remove key/secret above ## spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
