sollhui opened a new pull request, #68540:
URL: https://github.com/apache/doris/pull/68540
### What problem does this PR solve?
Problem Summary:
GCS access currently depends on HMAC credentials or unsigned requests. A
deployment using Application Default Credentials or a Compute Engine service
account cannot consistently use the same identity for object reads and writes,
Iceberg metadata operations, storage vaults, and backend error-log downloads.
Add native GCP OAuth authentication across FE, BE, and Cloud using
`gs.credential_provider_type=DEFAULT` or `COMPUTE_ENGINE`, with optional
`gs.impersonation_service_account`. GCP storage without explicit authentication
defaults to Application Default Credentials. Public-bucket access requires
explicit `ANONYMOUS`; existing HMAC configuration remains supported. Reject
conflicting native, HMAC, and AWS-only authentication properties before
creating clients.
The implementation follows the current master architecture:
- Keep credential resolution in the filesystem API, OAuth client
construction in the GCS plugin, and Iceberg client integration in the Iceberg
connector. Use the native Hadoop GCS connector for namespace operations and
bearer-authenticated S3 XML clients for object operations.
- Carry typed credentials through additive Thrift and Protobuf fields, Cloud
object information, and the shared C++ S3 client factory. Preserve AWS
authentication, client caching, and rate limiting.
- Normalize GCS aliases for resources, serialize resource ALTER operations,
and validate effective properties before publication. Preserve omitted vault
credential fields during partial ALTER, support clearing impersonation
explicitly, and reject anonymous storage vaults. Apply vault changes to
candidate metadata so a failed update does not persist a partial rename.
- Support `gs://` in EXPORT and OUTFILE while preserving HTTP paths.
Generate backend GCS V4 error-log URLs through IAM `signBlob`, falling back to
the local error-log path if signing fails.
- Add unit coverage and IAM regression suites for catalogs, TVFs, load,
export, outfile, backup/restore, resources, vaults, property validation, and
anonymous access. Export tests read back the written data and isolate cleanup
by run directory.
Example native authentication properties:
```properties
provider=GCP
gs.credential_provider_type=DEFAULT
gs.impersonation_service_account=tar...@project.iam.gserviceaccount.com
```
The impersonation property is optional. Use `COMPUTE_ENGINE` to select the
attached VM identity explicitly, or `ANONYMOUS` for public object access.
Compatibility and deployment:
- Existing AWS/HMAC protocol fields and authentication paths remain
available. Native GCP credentials require updated consumers; upgrade
MetaService and the relevant FE/BE processes before enabling the feature during
a rolling upgrade.
- Package google-cloud-cpp OAuth libraries for BE/Cloud and the shaded
Hadoop GCS connector for FE/Iceberg.
- FE presigned uploads with native OAuth remain unsupported and fail
explicitly. Backend error-log downloads use the dedicated GCS V4 signing path.
- Native OAuth uses the existing configurable object-store endpoint, so this
remains an outbound-request/SSRF surface. Catalog creation requires catalog
CREATE privilege; vault creation requires global ADMIN privilege. Google token
and signing operations use the credential library and Google IAM APIs. No
remote code-loading path is introduced.
### Release note
Add native GCP IAM authentication with Application Default Credentials,
Compute Engine identities, and optional service-account impersonation for
GCS-backed storage. GCP configurations without HMAC or an explicit
authentication mode now use ADC; public access must select `ANONYMOUS`. Improve
resource/vault credential updates and support GCS error-log signing.
### Validation
Passed locally:
- GCS filesystem reactor unit tests, including 88 filesystem API tests, 152
S3-base tests, and 36 GCS tests; zero failures, errors, or skips in these
modules.
- FE Core and Iceberg Checkstyle, plus the GCS reactor's style checks.
- clang-format 16 checks for all 33 changed C++ files, BE build hygiene,
connector import boundaries, and FE metadata-funnel checks.
- Parsing of all 41 changed Java files; Protobuf descriptor generation and
Thrift C++ generation for the changed definitions.
- `git diff --check`.
Not completed locally:
- Full FE unit tests: the standard runner requires JDK 17; the available
test runtime is JDK 20.
- Iceberg reactor tests: generation stops because the installed Thrift
compiler is 0.16.0 while master requires 0.24.0. The standalone Thrift syntax
check above does not replace this build.
- BE/Cloud compilation, unit tests, and clang-tidy: the installed thirdparty
bundle lacks the required google-cloud-cpp libraries.
- Live IAM regression suites: no configured Doris cluster or GCP test
environment was available.
### Check List (For Author)
- Test:
- [x] Unit tests added; filesystem API/S3/GCS tests executed successfully.
- [x] Regression tests added.
- [ ] Full FE, Iceberg, BE/Cloud, and live IAM tests completed.
- Behavior changed:
- [x] Yes: native GCP authentication, ADC default, explicit anonymous
mode, and validated partial credential updates as described above.
- Does this need documentation?
- [x] Yes: document credential properties, ADC deployment, impersonation
permissions, explicit anonymous access, and upgrade requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
