msabiransari opened a new issue, #18744:
URL: https://github.com/apache/iceberg/issues/18744
### Feature Request / Improvement
**Problem.** `iceberg-aws` declares the AWS SDK as `compileOnly` and the
docs tell users to bring either the SDK bundle or "individual AWS client
packages (Glue, S3, DynamoDB, KMS, STS)" (`docs/docs/aws.md`, "Enabling AWS
Integration"). For `S3FileIO` alone, the set that is actually required at
runtime is `s3` **plus `kms` and `sts`**; `glue` and `dynamodb` are not needed.
With only `s3` (and an HTTP client) on the classpath, `S3FileIO.initialize`
fails before any S3 call:
```
java.lang.NoClassDefFoundError:
software/amazon/awssdk/services/kms/model/EncryptionAlgorithmSpec
at org.apache.iceberg.aws.AwsProperties.<clinit>(AwsProperties.java:231)
at
org.apache.iceberg.aws.AwsClientFactories$DefaultAwsClientFactory.<init>(AwsClientFactories.java:101)
at
org.apache.iceberg.aws.AwsClientFactories.<clinit>(AwsClientFactories.java:50)
```
Add `kms` and it becomes:
```
java.lang.NoClassDefFoundError: software/amazon/awssdk/services/sts/model/Tag
at org.apache.iceberg.aws.AwsProperties.toStsTags(AwsProperties.java:487)
at org.apache.iceberg.aws.AwsProperties.<init>(AwsProperties.java:302)
at
org.apache.iceberg.aws.AwsClientFactories$DefaultAwsClientFactory.initialize(AwsClientFactories.java:179)
```
Add `sts` and initialization succeeds; `glue` and `dynamodb` make no
difference. Verified on 1.12.0 with SDK 2.54.17 (JDK 21); `AwsProperties`,
`AwsClientFactories` and the `iceberg-aws` build block are unchanged on `main`
(2742348d).
**Cause.** `S3FileIO.initialize` -> `S3FileIOAwsClientFactories.initialize`
falls back to `AwsClientFactories.from(...)` when `s3.client-factory-impl` is
unset -> `AwsClientFactories.<clinit>` builds `DefaultAwsClientFactory`, whose
constructor and `initialize` eagerly build `AwsProperties`. `AwsProperties` has
a static field of type `kms.model.EncryptionAlgorithmSpec` and its constructor
calls `toStsTags`, so the KMS and STS model classes are loaded even though the
S3-only path never uses them.
This is a smaller, precise instance of #7516 (avoid loading all AWS client
dependencies): the S3-only factory from #7590 helps only when
`s3.client-factory-impl` is set explicitly; the default path still pulls KMS
and STS.
**Reproduction.** Classpath: `org.apache.iceberg:iceberg-aws:1.12.0`,
`iceberg-core:1.12.0`, `software.amazon.awssdk:s3:2.54.17`,
`software.amazon.awssdk:url-connection-client:2.54.17`; no `kms`, no `sts`.
```java
import java.util.Map;
import org.apache.iceberg.aws.s3.S3FileIO;
public class InitS3FileIO {
public static void main(String[] args) {
try (S3FileIO io = new S3FileIO()) {
io.initialize(Map.of());
System.out.println("initialize OK");
} catch (Throwable t) {
System.out.println("initialize FAILED: " + t);
t.printStackTrace(System.out);
}
}
}
```
**Proposal.**
1. Build `AwsProperties` lazily in `DefaultAwsClientFactory`: keep the
property map, construct `AwsProperties` on first use in `glue()`, `kms()` and
`dynamo()` (3 call sites plus a private accessor; about 15 changed lines in one
class). With that change the program above prints `initialize OK` on the
S3-only classpath, and a real `exists()` call builds the S3 client and fails
only with `ConnectException` against a closed port. Prototype diff below
(compiled against the released jars and shadowed on the classpath; not yet run
through the project's own test suite / Spotless).
2. Correct `docs/docs/aws.md`: for `S3FileIO` the minimal set is `s3` + an
HTTP client today (plus `kms`/`sts` until (1) lands); Glue and DynamoDB are
only needed for `GlueCatalog` / `DynamoDbCatalog` / the DynamoDB lock manager.
Anything that constructs `AwsProperties` directly (`AwsKeyManagementClient`,
assume-role factories, SigV4 REST auth) legitimately still needs KMS/STS; (1)
only stops the S3-only default path from loading them. I can open the PR for
both parts; a regression test would load `S3FileIO` through a class loader that
refuses `software.amazon.awssdk.services.kms.*` and `sts.*`.
<details>
<summary>Prototype patch against the 1.12.0 sources</summary>
```diff
--- a/aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java
+++ b/aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java
@@ -23,6 +23,7 @@
import org.apache.iceberg.aws.s3.S3FileIOProperties;
import org.apache.iceberg.common.DynConstructors;
import org.apache.iceberg.relocated.com.google.common.base.Strings;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.util.PropertyUtil;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
@@ -92,18 +93,27 @@
}
static class DefaultAwsClientFactory implements AwsClientFactory {
- private AwsProperties awsProperties;
+ private Map<String, String> properties = ImmutableMap.of();
+ private volatile AwsProperties awsProperties;
private AwsClientProperties awsClientProperties;
private S3FileIOProperties s3FileIOProperties;
private HttpClientProperties httpClientProperties;
DefaultAwsClientFactory() {
- awsProperties = new AwsProperties();
awsClientProperties = new AwsClientProperties();
s3FileIOProperties = new S3FileIOProperties();
httpClientProperties = new HttpClientProperties();
}
+ private AwsProperties awsProperties() {
+ AwsProperties result = awsProperties;
+ if (result == null) {
+ result = new AwsProperties(properties);
+ awsProperties = result;
+ }
+ return result;
+ }
+
@Override
public S3Client s3() {
return S3Client.builder()
@@ -146,7 +156,7 @@
return GlueClient.builder()
.applyMutation(awsClientProperties::applyClientRegionConfiguration)
.applyMutation(httpClientProperties::applyHttpClientConfigurations)
- .applyMutation(awsProperties::applyGlueEndpointConfigurations)
+ .applyMutation(awsProperties()::applyGlueEndpointConfigurations)
.applyMutation(awsClientProperties::applyClientCredentialConfigurations)
.applyMutation(awsClientProperties::applyRetryConfigurations)
.build();
@@ -157,7 +167,7 @@
return KmsClient.builder()
.applyMutation(awsClientProperties::applyClientRegionConfiguration)
.applyMutation(httpClientProperties::applyHttpClientConfigurations)
- .applyMutation(awsProperties::applyKmsEndpointConfigurations)
+ .applyMutation(awsProperties()::applyKmsEndpointConfigurations)
.applyMutation(awsClientProperties::applyClientCredentialConfigurations)
.applyMutation(awsClientProperties::applyRetryConfigurations)
.build();
@@ -169,14 +179,16 @@
.applyMutation(awsClientProperties::applyClientRegionConfiguration)
.applyMutation(httpClientProperties::applyHttpClientConfigurations)
.applyMutation(awsClientProperties::applyClientCredentialConfigurations)
- .applyMutation(awsProperties::applyDynamoDbEndpointConfigurations)
+
.applyMutation(awsProperties()::applyDynamoDbEndpointConfigurations)
.applyMutation(awsClientProperties::applyRetryConfigurations)
.build();
}
@Override
public void initialize(Map<String, String> properties) {
- this.awsProperties = new AwsProperties(properties);
+ // AwsProperties references KMS and STS types; build it only when a
non-S3 client is requested
+ this.properties = properties;
+ this.awsProperties = null;
this.awsClientProperties = new AwsClientProperties(properties);
this.s3FileIOProperties = new S3FileIOProperties(properties);
this.httpClientProperties = new HttpClientProperties(properties);
```
</details>
### Query engine
Other
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
