xylaaaaa opened a new pull request, #69605:
URL: https://github.com/apache/doris/pull/69605
Issue Number: None
Related PR: #69082
Problem Summary: Every BE object storage entry (FE properties, cold-storage
resources, storage vaults) produced one flat `S3Conf` / `S3ClientConf` that
carried all providers in a single shape: Azure reused ak/sk/bucket as
account/key/container, while region, role_arn and the credential provider type
only meant something for S3. `S3ClientFactory::create` then branched on the
provider to build an S3 or an Azure client, and `ObjClientHolder` merged
credential refreshes field by field into that flat struct. A backend-specific
field could leak into the other backend's identity or validation (for example,
an Azure configuration still needed a region), and adding an Azure
authentication method would have meant more flat fields that S3 must ignore.
This is the first step of splitting #69082 (BE object storage refactor and
native Azure SAS / OAuth2 access). It only restructures the BE; no FE change,
no Thrift change, and Azure still supports only SharedKey.
- Typed configuration. `io::ObjClientConfig` holds the shared connection
options, the client usage (internal storage / external access) and exactly one
backend, `std::variant<S3ClientConfig, AzureClientConfig>`. The Azure
authentication is a variant as well, with SharedKey as its only alternative for
now. `io::ObjStorageConfig` (bucket, prefix, client) replaces `S3Conf`;
`S3Conf` and `S3ClientConf` are removed.
- One conversion point. `ObjStorageConfigResolver::from_properties`
(formerly `S3ClientFactory::convert_properties_to_s3_conf`),
`from_storage_param` and `from_object_store_info` (formerly
`S3Conf::get_s3_conf`) turn every source into the typed configuration. The flat
field mapping and its validation are ported unchanged. `merge_update` merges a
refreshed resource or vault configuration with the same field rules as the flat
merge.
- Client construction by provider. `ObjClientFactory` validates every
request with the rules of its backend (the checks `S3ClientFactory::create`
applied) and dispatches through `std::visit`, so a backend without a branch
fails to compile. `S3ClientProvider` and `AzureClientProvider` build and cache
their clients outside the cache lock, and one shared function applies the BE
rate limiter. `ObjClientFactory` also owns the test hook
`set_client_creator_for_test`. `S3ClientFactory` keeps only the AWS SDK
lifetime and the AWS helpers (client configuration, credentials provider, CA
path), which Kinesis also uses.
- Holder. `ObjClientHolder` moves to
`io/obj_storage/obj_client_holder.{h,cpp}` and owns an immutable
`ObjClientState` (config, client, generation) that it replaces as a whole, so a
reader never pairs a new config with an old client.
- Locations. `ObjStorageUri::parse_s3` replaces `S3URI` with the same rules.
`FileFactory` resolves FILE_S3 locations through one function for the reader,
`create_fs` and `create_file_writer`; the other callers (export, snapshot,
connectivity test, cold storage, storage vaults, AI embed, Kinesis, benchmarks,
spill) use the resolver.
- GCP IAM credentials (#68540) move with the S3 fields:
`S3AuthConfig::credential` carries the provider-native `CredentialConfig`, the
resolver parses and converts it for all three sources and validates it as
before, `ObjClientFactory` validates it on every request, and
`S3ClientProvider` builds the client through `make_s3_client` with the GCS V4
signed URL generator. A reset takes the vendor and credential from the update,
as the flat merge does.
- The AWS SDK RFC 3986 encoding flag is set once at SDK initialization
instead of in every reader and writer constructor, and the reader no longer
uses AWS types for HTTP 429.
Behavior changes:
1. The S3 client cache no longer keys on the bucket: different buckets with
the same endpoint, credentials and connection options share one client.
Building an S3 client never used the bucket.
2. A holder reset merged onto a state that another reset already replaced
returns `Aborted` instead of overwriting the newer update; the caller retries
with its next refresh.
3. A holder can no longer be reset to another backend (InvalidArgument),
which is a different storage target.
4. A flat Azure configuration no longer needs a region (FE sent a
placeholder).
None
- Test: Unit Test (new obj_client_factory_test and obj_storage_uri_test, the
latter porting the S3URI cases; existing users, including the GCP IAM cases of
s3_client_factory_test, migrated, with test configs built through
testutil/obj_storage_test_util.h). Not run locally; run in CI.
- Behavior changed: Yes, as listed above
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
