xylaaaaa opened a new pull request, #69667:
URL: https://github.com/apache/doris/pull/69667
> **Depends on #69605.** This PR is stacked on #69605 (BE-1 of the #69082
split), so until #69605 is merged its diff also contains the #69605 commits.
Only the last commit (`[refactor](be) Rename the common object storage IO
classes to ObjStorage*`) belongs to this PR. It will be rebased onto master
after #69605 is merged.
### What problem does this PR solve?
Issue Number: None
Related PR: #69082, #69605
Problem Summary: This is BE-2 of the split of #69082 and is based on #69605.
After #69605 the
common object storage IO layer serves both S3 and Azure, but its classes,
files and several
identifiers still carry S3 names. This change renames them; behavior is
unchanged.
1. `S3FileSystem` / `S3FileReader` / `S3FileWriter` become
`ObjStorageFileSystem` /
`ObjStorageFileReader` / `ObjStorageFileWriter`, and
`io/fs/s3_file_{system,reader,writer}.*`
become `io/fs/obj_storage_file_{system,reader,writer}.*`.
`s3_file_bufferpool.*` becomes
`obj_storage_file_buffer_pool.*` (file names only). The unity skip list
and the header hygiene
tables follow the new file names.
2. `FileSystemType::S3` becomes `FileSystemType::OBJECT_STORAGE` with the
same value 1, and is
printed as `OBJECT_STORAGE`.
3. `ObjClientHolder::full_s3_path` and `ObjStorageFileSystem::full_s3_path`
become `display_path`.
4. `S3ClientFactory` becomes `AwsSdkManager`: after #69605 it no longer
builds clients and only
owns the AWS SDK lifetime and the AWS helpers that object storage clients
and Kinesis share.
5. `FileWriterOptions::used_by_s3_committer` becomes `deferred_completion`.
Whether the
coordinator can complete a staged upload is now described by
`ObjStorageCapabilities::deferred_completion` (`ANY_COORDINATOR` for S3
multipart,
`NAMESPACED_BLOCK_IDS` for Azure block blobs) instead of comparing the
provider with AZURE.
`VHivePartitionWriter` still rejects an Azure write from a coordinator
without
`supports_deferred_azure_multipart`.
6. In `CachedRemoteFileReader`, the S3-specific names of the
peer-versus-remote read path
(`launch_s3_race`, `do_s3_read`, `RaceState::s3_done`,
`_execute_s3_fallback`, ...) become
`remote`, and the free function `execute_s3_read` becomes
`read_remote_span`.
7. File-local bvar variables in the reader and writer are renamed to
`obj_storage_*`.
Exposed names are kept for compatibility: bvar and metric names,
`DorisMetrics` members, config
names, thread pool names, debug point and sync point names (regression and
unit tests trigger
them by name), error message texts, and gtest suite and test file names.
### Release note
None
### Check List (For Author)
- Test: Unit Test (updated
SpillIcebergTableSinkOperatorTest.AzureDeferredMultipartRequiresCoordinatorCapability);
not compiled or run locally, CI compiles and runs BE UT
- build-support/check-build-hygiene.sh: header layering, closure,
pairing and unity-skip checks pass;
the util/threadpool.h reach budget failure already exists on #69605
and is fixed on master
- Behavior changed: No (the printed name of FileSystemType::OBJECT_STORAGE
in logs changes from S3)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
