cyriltovena opened a new pull request, #65433:
URL: https://github.com/apache/doris/pull/65433
### What problem does this PR solve?
Issue Number: close #65432
Related PR: #62788, #64766 (the vault credentials-provider plumbing this
builds on), #62987 (Google auth for Iceberg REST catalogs, FE-side)
Problem Summary:
Storage vaults with `provider = GCP` can only authenticate with static HMAC
keys today — the existing keyless mechanisms (`s3.role_arn`, the
`s3.credentials_provider_type` values) are all AWS-semantic. On GKE, where
workloads authenticate via Workload Identity / Application Default Credentials,
this forces users to mint and distribute long-lived HMAC key pairs.
This PR adds a `GCP_ADC` credentials provider type for GCP vaults. The GCS
XML API accepts `Authorization: Bearer <oauth2-token>` as an alternative to
HMAC signatures, so no new SDK dependency is introduced:
```sql
CREATE STORAGE VAULT gcs_vault PROPERTIES (
"type" = "S3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "us-central1",
"s3.bucket" = "my-bucket",
"s3.root.path" = "doris",
"provider" = "GCP",
"s3.credentials_provider_type" = "gcp_adc"
);
```
Design (following the seams introduced by #64766 as closely as I could):
- **Proto/Thrift**: new `CredProviderTypePB::GCP_ADC = 9` /
`TCredProviderType::GCP_ADC`.
- **common/cpp**: a shared `GcpAdcTokenProvider` resolves OAuth2 tokens in
ADC order — `GOOGLE_APPLICATION_CREDENTIALS` service account json (RS256 JWT
assertion via OpenSSL, exchanged at the token endpoint), else the GCE/GKE
metadata server (Workload Identity). Tokens are cached and refreshed ahead of
expiry; built on the AWS SDK's own HTTP/JSON utilities so both BE and the
meta-service recycler can use it.
- **BE / recycler**: for `GCP_ADC` the S3 client is created with
`AnonymousAWSCredentialsProvider` (SigV4 signing is skipped on empty
credentials) and `S3ObjStorageClient` / `S3ObjClient` attach the bearer token
per request via `SetAdditionalCustomHeaderValue`. Re-applied per page in
paginated listings so token rotation mid-listing is safe.
- **FE**: `s3.credentials_provider_type = "gcp_adc"` is validated (provider
GCP only; mutually exclusive with ak/sk and role_arn) and mapped in
`S3Properties.getObjStoreInfoPB`. The FE connectivity probe is skipped for this
mode (its Java client signs with static credentials); the BE probes the vault
at startup instead, same as assumed-role vaults.
- **Meta-service**: `ADD_S3_VAULT` accepts keyless GCP vaults with
`GCP_ADC`; `ALTER_S3_VAULT` supports switching an HMAC vault to `GCP_ADC`
(clearing stored keys) and back, symmetric with the existing ak/sk ⇄ role_arn
switches.
Known limitation: presigned URLs are not supported in this mode (a
short-lived bearer token cannot be embedded in a URL); `generate_presigned_url`
logs a warning and returns an empty string.
This is my first contribution to Doris, so I fully expect there are
conventions I've missed — very happy to rework any part of the approach
(naming, where the token provider lives, whether a native GCS client would be
preferred longer-term) based on your feedback.
### Release note
Support keyless GCS authentication for storage vaults via Google Application
Default Credentials / Workload Identity (`"s3.credentials_provider_type" =
"gcp_adc"`).
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- Created a `GCP_ADC` vault on a GKE cluster with Workload Identity
enabled (metadata-server path) and verified table create/load/query/compaction
and recycler GC against the GCS bucket, plus `ALTER STORAGE VAULT` switching
between HMAC keys and `gcp_adc`.
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] Yes. I will open a doris-website PR documenting the new `gcp_adc`
value on the CREATE/ALTER STORAGE VAULT pages once the property surface is
settled in review.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
