n-dlms commented on issue #7516:
URL: https://github.com/apache/iceberg/issues/7516#issuecomment-5189179283
## Status check + a concrete path to close this out
I've been looking at this issue and wanted to share a status check of what
already exists, where the actual gap is, and a concrete proposal to finish it.
Hopefully this helps the issue move forward.
**What's already merged (the first half of this issue)**
- `S3FileIOAwsClientFactory` (S3-only factory interface) +
`s3.client-factory-impl` catalog property, added in #7590 (merged 2023-05-16).
- `S3FileIOAwsClientFactories.initialize()`
(`aws/src/main/java/org/apache/iceberg/aws/S3FileIOAwsClientFactories.java:41`)
loads the factory from `s3.client-factory-impl` and falls back to
`AwsClientFactories.from()` when unset — so backward compatibility holds.
- `S3FileIO` (`S3FileIO.java:511`) and `PrefixedS3Client` both route through
it.
So the *factory* half of "if I just want S3FileIO, I shouldn't need
glue/dynamodb" is done at the class level: `iceberg-aws` itself declares all
AWS SDK deps as `compileOnly` (`build.gradle:526-539`), meaning the non-bundle
artifact doesn't drag the other services in transitively.
**Where the gap actually is: the deployment artifact**
The problem the issue describes ("users dump the bundle jar, 450MB") lives
in **`iceberg-aws-bundle`**, which is still the artifact the docs tell everyone
to add:
- `docs/docs/aws.md:50,688` — `--packages ...
org.apache.iceberg:iceberg-aws-bundle:...`
- `docs/docs/aws.md:72-77` — wget of `iceberg-aws-bundle-<version>.jar`
and the bundle currently shades **everything**
(`aws-bundle/build.gradle:36-48`):
```
s3accessgrants, apache-client, auth, crt-core, http-auth-aws-crt,
iam, sso, s3, kms, glue, sts, dynamodb, lakeformation, analyticsaccelerator
```
So an S3-only user who follows the docs ships
Glue/DynamoDB/KMS/LakeFormation/STS/IAM/SSO + the CRT + analytics-accelerator
in their runtime classpath. That's the ~450MB. The factory split (#7590)
doesn't fix this because the fat bundle is what's deployed.
**Concrete proposal (smallest first step)**
Split the bundle along the same line the factories were split:
1. **Slim `iceberg-aws-bundle`**: keep only the S3-path dependencies (s3,
auth, apache-client, crt-core/http-auth-aws-crt, url-connection-client,
s3accessgrants, analyticsaccelerator) — everything the
`S3FileIOAwsClientFactory` path needs.
2. **New `iceberg-aws-bundle-full`** (or `iceberg-aws-catalog-bundle`): the
current content, for GlueCatalog/DynamoDB locks/KMS users.
3. Docs updated to recommend the slim bundle for S3FileIO use and the full
bundle for catalog/lock use.
This is additive (no breaking change: the slim bundle's package surface
keeps `AwsClientFactories`/`AwsProperties` for fallback users), and it directly
removes the "450MB for S3" pain the issue calls out. The Glue/DynamoDB classes
simply move to the full artifact; anyone who needs them changes one artifact
name.
**Open questions for maintainers**
1. Is splitting the bundle the direction the community wants, or is a
slimmer default bundle (with the catalog clients only in a separate artifact)
preferred?
2. Naming: `iceberg-aws-bundle-full` vs `iceberg-aws-catalog-bundle` vs
keeping `iceberg-aws-bundle` full and adding `iceberg-aws-bundle-s3`?
3. Is there appetite for a compatibility shim (deprecated artifact that
depends on the full bundle) for users currently consuming `iceberg-aws-bundle`
for GlueCatalog?
4. Does the release process handle an additional bundle artifact cleanly
(license/NOTICE/`runtime-deps`)?
Happy to prototype the slim bundle + docs update if this direction is right
— it's a self-contained change with an easy size comparison as evidence.
---
**AI Disclosure**
- Model: Claude Opus 4.7
- Platform/Tool: Claude Code
- Human Oversight: fully reviewed
- Prompt Summary: Analyze issue #7516 status, identify the remaining gap
(bundle artifact), propose a split-bundle solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
