Simanchal-Liveramp opened a new issue, #18880:
URL: https://github.com/apache/iceberg/issues/18880
### Feature Request / Improvement
**Problem**
We run Spark jobs on Kubernetes (EKS) that authenticate to an Iceberg REST
catalog (Databricks Unity Catalog's Iceberg REST endpoint) using workload
identity. Each job has a short-lived subject token, the pod's projected service
account token, which is exchanged at the catalog's OIDC endpoint (RFC 8693) for
a short-lived access token. We want this without distributing a client secret
to the job, and with token lifetimes kept short for security reasons.
Existing options don't fit this pattern:
- `token` is a static value. Long-running jobs fail once it expires.
- `rest.auth.type=oauth2` with `credential` requires a client ID and secret,
which we're trying to avoid.
- `BearerAuthManager` (#18754) reads the token from a file on a fixed reload
interval. Iceberg doesn't check token expiry, so refresh timing depends on an
external producer writing the file ahead of expiry. Iceberg never calls that
producer, and a stale file causes requests to fail.
**Requested capability**
Let users configure a token provider class that Iceberg calls to obtain a
bearer token. Iceberg should own the refresh schedule: refresh a configurable
margin before `expiresAt`, retry on failure while keeping the current token,
invalidate and fetch again on a 401, and share one token across sessions and
tables in a catalog.
**Proposed interface**
```java
public interface BearerTokenProvider extends AutoCloseable {
/** Returns a valid access token and its expiry. Called on startup, before
expiry, and after a 401. */
BearerToken fetchToken() throws IOException;
}
public record BearerToken(String value, Instant expiresAt) {}
```
Configuration (names are a suggestion):
```
rest.auth.type=bearer
rest.auth.bearer.token-provider=com.example.TokenExchangeProvider
rest.auth.bearer.token-provider.<key>=<value> # passed to the provider
```
**Acceptance criteria**
- Refresh happens before `expiresAt` minus a configurable margin, not on a
fixed interval.
- If a refresh fails, the current token stays in use and a retry is
scheduled.
- A 401 triggers one immediate re-fetch.
- One token is shared by all sessions and tables in a catalog.
- Provider configuration is passed through, and Iceberg needs no secrets in
its own config.
- Tests cover refresh timing, retry on failure, and concurrent access.
**Use cases**
- Kubernetes projected service account token exchanged with the catalog's
identity provider
- Cloud workload identity (AWS, GCP, Azure) exchanged for a catalog access
token
- Platform-issued federation tokens for any REST catalog that supports token
exchange
**Related:** #18754 (file-based BearerAuthManager), #13550 (original
bearer-from-file request)
**Environment:** Iceberg 1.9+, Spark (version), Scala (version), Kubernetes
(EKS)
**Contribution:** I can contribute a patch if the design is acceptable.
---
### Query engine
Spark
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
