morningman opened a new pull request, #61919:
URL: https://github.com/apache/doris/pull/61919
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Refactor the Apache Doris FE filesystem layer so that each storage backend
(S3, OSS, COS, OBS, Azure, HDFS, Local) lives in its own independent Maven
module and is discovered at runtime via Java ServiceLoader (SPI pattern).
Adding a new storage backend now requires zero changes to fe-core.
Changes by step:
**P3.0b** – Rename `FileSystemProvider` → `LegacyFileSystemProviderFactory`
to free up the name `FileSystemProvider` for the new SPI interface.
**Step 1** – Create `fe/fe-filesystem/` aggregator POM and
`fe-filesystem-spi`
module. Defines the stable SPI contracts:
- `FileSystem`, `ObjStorage`, `FileSystemProvider` interfaces
- `HadoopAuthenticator`, `IOCallable` for HDFS/Kerberos
- Value objects: `FileEntry`, `FileIterator`, `InputFile`, `OutputFile`,
`RequestBody`, `ObjectMetadata`, `MultipartPart`, `UploadedPart`
**Step 2** – `fe-filesystem-s3` module:
- `S3Uri`, `S3ObjStorage`, `S3FileSystem`, `S3OutputStream`
- `S3FileSystemProvider` registered via META-INF/services
**Step 3** – `fe-filesystem-oss`, `fe-filesystem-cos`, `fe-filesystem-obs`:
- Thin providers that reuse S3FileSystem via S3-compatible APIs
- Each translates vendor-specific property keys to AWS_* keys
**Step 3b** – `fe-filesystem-azure` module:
- `AzureUri` (wasb/wasbs/abfs/abfss/https), `AzureObjStorage`
- Supports shared-key, service-principal, and default-credential auth
- Multipart upload via Azure block-blob stageBlock/commitBlockList
**Step 4** – `fe-filesystem-hdfs` module:
- `HdfsConfigBuilder`, `SimpleHadoopAuthenticator`,
`KerberosHadoopAuthenticator`
- `DFSFileSystem` implements `spi.FileSystem` directly
- Supports hdfs/viewfs/ofs/jfs/oss schemes
**Step 6** – `fe-filesystem-local` (test-only):
- `LocalFileSystem` using `java.nio.file` for unit-test isolation
**Step 7** – `fe-core` integration:
- `fe-filesystem-spi` added as compile dep; all impl modules as runtime
deps
- New `StoragePropertiesConverter` translates `StorageProperties` → Map
- `FileSystemFactory` extended with `getFileSystem(Map)` /
`getFileSystem(StorageProperties)` using ServiceLoader (double-checked
locking); old `get()` methods kept `@Deprecated` for Phase-4 migration
- Fix checkstyle violations (import order, line length) in HMS classes
introduced by P3.0b rename
### Release note
None (internal refactoring; no user-visible behavior change)
### Check List (For Author)
- Test: `mvn install -pl fe-core -am -DskipTests` passes (BUILD SUCCESS)
- Behavior changed: No (legacy API fully preserved; new SPI API added
alongside)
- Does this need documentation: No
Co-authored-by: Copilot <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
