laxmanmamididd opened a new pull request, #7363:
URL: https://github.com/apache/datafusion-comet/pull/7363
## Which issue does this PR close?
No issue filed yet; I can open one if you'd like this tracked separately.
## Rationale for this change
The native Iceberg scan signs S3 requests with the `s3.*` credentials an
Iceberg REST catalog vended when the driver loaded the table. Those credentials
are short-lived (about an hour on Unity Catalog), so a scan still running after
they expire fails with `ExpiredToken`. Vanilla Spark on the same table keeps
working, because Iceberg's `S3FileIO` refreshes the credentials from the
endpoint the catalog returns in `client.refresh-credentials-endpoint`.
The S3 credential SPI (#4309, #6509) lets a provider do the same, and the
docs point at `IcebergRESTVendedS3Provider` as the reference to copy. That
reference builds `VendedCredentialsProvider` straight from the FileIO
properties, which needs `credentials.uri`. A REST catalog returns the endpoint
under `client.refresh-credentials-endpoint`, and `S3FileIO` reaches it through
`AwsClientProperties`, which maps it to the URI and also carries the catalog
URI and OAuth properties. Copied as is, the reference doesn't refresh against a
real REST catalog. Its test sets `credentials.uri` by hand and exercises only
the static path.
## What changes are included in this PR?
- `IcebergVendedS3CredentialProvider` (Spark 4.x, main scope). `initialize`
hands the FileIO bag to Iceberg's
`AwsClientProperties.credentialsProvider(...)` and gets back the same provider
`S3FileIO` would use:
- a `VendedCredentialsProvider` when the catalog returned a refresh
endpoint;
- the static vended keys otherwise;
- the default chain when neither is set.
`getCredentialsForPath` reports the session expiry, so Comet's credential
reuse (#6509) stops before expiry and asks again. Iceberg is loaded by
reflection from the context class loader, as elsewhere in Comet, so Comet gains
no compile-time Iceberg dependency.
- Docs: a section for the provider in `s3-credential-providers.md`, and a
note on the reference implementation explaining the `credentials.uri` gap.
Activation:
```
spark.sql.catalog.<catalog>.s3.comet.credential.provider.class=org.apache.comet.cloud.s3.IcebergVendedS3CredentialProvider
```
## How are these changes tested?
`IcebergVendedS3CredentialProviderTest` (6 tests) runs a local HTTP server
standing in for the catalog's `/v1/credentials` endpoint, so refresh goes
through Iceberg's real `VendedCredentialsProvider`. It covers:
- expired vended credentials refreshed through
`client.refresh-credentials-endpoint`, with the refreshed expiry reported;
- unexpired credentials reused without calling the endpoint;
- the static vended keys when the catalog returns no endpoint;
- `client.refresh-credentials-enabled=false` honored;
- static keys without a session token, reported with no token;
- use before `initialize`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
