parthchandra opened a new pull request, #6025:
URL: https://github.com/apache/datafusion-comet/pull/6025
## Which issue does this PR close?
Closes #6024
## Rationale for this change
On EKS/IRSA, a temporary STS throttle during startup was becoming a
permanent failure: the
credential chain didn't retry, fell back to the node role that can't read
the bucket, and the whole
job died with 403s. This makes the native scan safe under a large concurrent
startup instead of
relying on luck (fewer cores, slower ramp).
## What changes are included in this PR?
A new Comet-owned web-identity credential provider
(`native/core/src/cloud/s3/web_identity.rs`)
that both native scan paths use when IRSA is detected and no credentials are
configured explicitly.
It:
- **Retries the throttle.** Uses the AWS SDK web-identity provider, whose
STS client retries
throttling with backoff and jitter. The attempt count is configurable
(default 5).
- **Never downgrades.** It's web-identity only — no node/instance-role
fallback — so a throttle that
outlasts the retries returns an error instead of a wrong-identity
credential.
- **Shares one credential per process.** Cached and keyed by (role, token
file, region), shared
across all reader threads and scans, refreshed ahead of expiry with a
per-process jitter so
cluster-wide refreshes don't sync up.
Wiring and precedence (highest first):
1. An explicit Comet credential provider class -> unchanged, uses the
existing JVM bridge.
2. Explicit credentials in config (static keys, `client.assume-role.arn`, or
`aws.credentials.provider`) -> left alone.
3. IRSA detected and nothing explicit set -> the new provider.
4. Otherwise -> the existing default chain.
So this only changes the previously-default behavior; anyone with explicit
credentials is
unaffected. It's on by default under IRSA and can be turned off with
`comet.s3.credentials.webIdentity.enabled=false`. Knobs (`maxAttempts`,
`minTtlSeconds`,
`refreshJitterSeconds`) are read from the `fs.s3a.*` config (Parquet) or the
catalog properties
(Iceberg).
Files touched:
- New: `native/core/src/cloud/s3/web_identity.rs`
- `iceberg_common.rs` — install the provider on the Iceberg path (with a
guard that defers to
explicit credentials).
- `s3.rs` — install it on the Parquet path when no explicit provider is set.
- Docs: user guide (new "EKS / IRSA" section) and the credential-provider
design notes.
## How are these changes tested?
New Rust unit tests in `web_identity.rs` (run offline, no network):
- **Detection and config:** IRSA needs both env vars; the enable toggle
works; knobs parse with
fallback; explicit credentials cause the take-over to stand aside.
- **Caching:** a fresh credential is reused, a near-expiry one is refreshed,
and a burst of
concurrent readers triggers exactly one STS call.
- **Throttle behavior (the fix):** using an in-memory HTTP client that
mimics STS —
- transient throttle (throttle, throttle, success) -> retried, exactly 3
attempts, returns real
credentials;
- persistent throttle -> returns an error, bounded to `maxAttempts`, never
a downgraded
credential.
All tests pass; clippy and rustfmt are clean.
Not covered here (needs a network-capable CI runner or a cluster): an
end-to-end test against a
live/mock STS, and a rerun of the original high-concurrency workload. A
socket-based mock can't run
in the local sandbox, so the retry/no-downgrade logic is covered by the
in-memory HTTP client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
