AwasthiSomesh opened a new issue, #10078:
URL: https://github.com/apache/iceberg/issues/10078
Hi Team ,
We are doing below code to access iceberg table from glue catalog and data
storage as S3
var spark = SparkSession.builder().master("local[*]")
.config("spark.sql.defaultCatalog", "AwsDataCatalog")
.config("spark.sql.catalog.AwsDataCatalog",
"org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.AwsDataCatalog.catalog-impl",
"org.apache.iceberg.aws.glue.GlueCatalog")
.config("spark.sql.catalog.AwsDataCatalog.io-imp",
"org.apache.iceberg.aws.s3.S3FileIO")
.config("spark.hadoop.fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXXXXxxx")
.config("spark.hadoop.fs.s3a.secret.key",
"XXXXXXXXXXXXXXXXXXXXXXXXXXXx")
.config("spark.hadoop.fs.s3a.aws.region", "us-west-2")
.getOrCreate();
val df1 = spark.sql("select * from default.iceberg_table_exercise1");
Error-
Exception in thread "main"
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from any of the providers in the chain
AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(),
EnvironmentVariableCredentialsProvider(),
WebIdentityTokenCredentialsProvider(),
ProfileCredentialsProvider(profileName=default,
profileFile=ProfileFile(profilesAndSectionsMap=[])),
ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) :
[SystemPropertyCredentialsProvider(): Unable to load credentials from system
settings. Access key must be specified either via environment variable
(AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).,
EnvironmentVariableCredentialsProvider(): Unable to load credentials from
system settings. Access key must be specified either via environment variable
(AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).,
WebIdentityTokenCredentialsProvider(): Either the environment variable
AWS_WEB_IDENT
ITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set.,
ProfileCredentialsProvider(profileName=default,
profileFile=ProfileFile(profilesAndSectionsMap=[])): Profile file contained no
credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]),
ContainerCredentialsProvider(): Cannot fetch credentials from container -
neither AWS_CONTAINER_CREDENTIALS_FULL_URI or
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.,
InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
This code is throwing unable to load access and secret key but when we
passing these information with System.setproperty then its working.
But our requirement is set to spark level not system level.
Jars we used : - iceberg-spark-runtime-3.5_2.12-1.5.0 ,
iceberg-aws-bundle-1.5.0
Please any one help ASAP
Thanks,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
