hgschmie opened a new issue, #10085: URL: https://github.com/apache/iceberg/issues/10085
### Apache Iceberg version 1.5.0 (latest release) ### Query engine Flink ### Please describe the bug 🐞 We have a test setup where multiple AWS profiles exist in the usual (~/.aws/config and ~/.aws/credentials) location. We use one set of credentials to connect to a local minio instance and a second set of credentials to connect to a remote MSK cluster. we use the following properties with the Flink Catalog loader: ``` S3FileIOProperties.ENDPOINT -> http://minio:9000 S3FileIOProperties.PATH_STYLE_ACCESS -> "true" AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER -> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" "client.credentials-provider.aws.profile" -> "local-minio" ``` However, the iceberg writer tries to use the "default" profile which does not connect correctly to minio. I traced the problem down to `AwsClientProperties#createCredentialsProvider(Class<?> providerClass)` which does this: ```java provider = DynMethods.builder("create") .hiddenImpl(providerClass, Map.class) .buildStaticChecked() .invoke(clientCredentialsProviderProperties); ``` (find a static `create` method in the configured AWS provider class and pass a map of all the `client.credentials-provider.` prefixed properties to the provider) However, `ProfileCredentialsProvider` has no such method. So iceberg instantiates the provider with an empty `create()` method (that exists) and uses the default region. It might be good to point this out in documentation. I did solve the problem for myself by using a custom provider: ```java public final class IcebergProfileCredentialsProvider implements AwsCredentialsProvider { private static final String AWS_PROFILE_NAME = "iceberg-aws.profile"; public static final String PROFILE_NAME = "client.credentials-provider." + AWS_PROFILE_NAME; private final ProfileCredentialsProvider delegate; /** * The iceberg AWS credential provider logic looks for a magic method with this signature... */ public static AwsCredentialsProvider create(Map<String, String> properties) { var profileName = properties.get(AWS_PROFILE_NAME); return (profileName == null) ? new IcebergProfileCredentialsProvider() : new IcebergProfileCredentialsProvider(profileName); } IcebergProfileCredentialsProvider() { this.delegate = ProfileCredentialsProvider.create(); } IcebergProfileCredentialsProvider(String profileName) { this.delegate = ProfileCredentialsProvider.create(profileName); } @Override public AwsCredentials resolveCredentials() { return delegate.resolveCredentials(); } @Override public Class<AwsCredentialsIdentity> identityType() { return delegate.identityType(); } @Override public CompletableFuture<AwsCredentialsIdentity> resolveIdentity(ResolveIdentityRequest request) { return delegate.resolveIdentity(request); } } ``` Using ``` AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER -> "IcebergProfileCredentialsProvider" "client.credentials-provider.iceberg-aws.profile" -> "local-minio" ``` as catalog properties makes this work for me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
