This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-fs-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard c2bcb6515f3 [refactor](fs) Phase4 caller migration (#62016)
discard 8535923f46c [refactor](fs-spi) Phase4 P4.0: add FileSystemTransferUtil
and unit tests (#61928)
add af439c31ef9 [refactor](fe) Phase4 P4.0: add FileSystemTransferUtil and
unit tests
add a7eef553d2a [refactor](fe) Phase4 P4.1: add FileSystemPluginManager
and plugin loading infrastructure
add 52ad56099bd [refactor](fe) Phase4 P4.2: migrate FileSystemCache and
callers to spi.FileSystem
add 2f4ab911679 [refactor](fe) Phase4 P4.5b: migrate InsertIntoTVFCommand
to spi.FileSystem
add c47dbae11c1 [refactor](fe) Phase4 P4.6e: replace instanceof checks
with FsStorageType in RepositoryMgr
add 2be1cfff89f [feature](cloud) Migrate cloud ObjFS callers to new
filesystem SPI (P4.3)
add 102fd861329 [feature](fe) Migrate Iceberg DelegateFileIO to filesystem
SPI (P4.4)
add 3d715e2cd9b [refactor](fe) Phase4 P4.6a-d: migrate backup domain to
FileSystemDescriptor
add cab565c27dd [refactor](fe) P4.5c: S3SourceOffsetProvider SPI migration
(globListWithLimit)
add 8b376a639da [refactor](fe) Migrate HMSTransaction from
LegacyFileSystemApi to spi.FileSystem (P4.7)
add f084d6636ac [feature](fe) Implement fe-filesystem-broker SPI module
and migrate BrokerUtil
add af258fe33d7 [feature](fe) Migrate Repository.java I/O path to SPI
FileSystem
add 6b56f77fe41 [refactor](fe) Remove PersistentFileSystem from Repository
metadata path (P4.6-meta)
add 7d0cae7ca33 [fix](fe) Fix FE build: update tests to use new
StorageProperties/SPI FileSystem API
add 09a18f920a5 [fix](fe) Add missing ASF license headers to SPI service
provider files
add 0ed619cd710 [fix](fe) Fix fe-filesystem plugin module build errors and
wire into build.sh
add a4bdd5f772f [fix](build) Fix dependency:copy-dependencies path for
fe-filesystem plugins
add 9eb581f05f5 [fix](build) Handle Maven Build Cache hit missing
target/lib in FE output copy
new 13dc273b565 [refactor](fs-spi) P4.8-A: delete LegacyToNewFsAdapter and
LegacyFileSystemAdapter
new 86486c85076 [refactor](fs-spi) P4.8-B: migrate DirectoryLister from
RemoteFile to SPI FileEntry
new 216454f1996 [refactor](fs-spi) P4.8-C: delete MultipartUploadCapable
interface
new 3e90dc92d0e [refactor](fs-spi) P4.8-D: migrate HdfsStorageVault to
SPI, delete legacy HDFS IO wrappers
new 9d539779e26 [refactor](fs-spi) P4.8-E: delete SwitchingFileSystem and
dead legacy provider infrastructure
new 59884ca553c [refactor](fs-spi) P4.8-F: delete legacy BrokerFileSystem,
S3FileSystem, AzureFileSystem and StorageTypeMapper
new a844c736ccc [refactor](fs-spi) P4.8-G1G2: delete dead DFSFileSystem
subclasses and legacy ObjFileSystem
new 8733d0e8013 [refactor](fs-spi) P4.8-G3: delete DFSFileSystem and its
phantom reference infrastructure
new c6eae8d0fbf [refactor](fs-spi) P4.8-G4G5G6H1H2: delete legacy
filesystem hierarchy
new 29a2c3d697f [doc](fs-spi) P4.8: update progress doc to reflect all
phases complete
new 53e2c974ff6 [fix](build) Fix fe-filesystem plugin build and deployment
new 9d1c831e807 [improvement](build) Flatten filesystem plugin output
layout
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (c2bcb6515f3)
\
N -- N -- N refs/heads/branch-fs-spi (9d1c831e807)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build.sh | 15 +-
.../java/org/apache/doris/backup/Repository.java | 10 -
.../org/apache/doris/catalog/HdfsStorageVault.java | 32 +-
.../apache/doris/datasource/ExternalCatalog.java | 9 +-
.../org/apache/doris/datasource/hive/AcidUtil.java | 46 +-
.../doris/datasource/hive/HMSExternalCatalog.java | 3 +-
.../datasource/hive/HiveExternalMetaCache.java | 37 +-
.../org/apache/doris/datasource/hive/HiveUtil.java | 5 -
.../java/org/apache/doris/fs/DirectoryLister.java | 4 +-
.../main/java/org/apache/doris/fs/FileSystem.java | 3 +-
.../org/apache/doris/fs/FileSystemDescriptor.java | 10 -
.../apache/doris/fs/FileSystemDirectoryLister.java | 13 +-
.../org/apache/doris/fs/FileSystemFactory.java | 44 --
.../java/org/apache/doris/fs/FileSystemLookup.java | 30 -
.../apache/doris/fs/FileSystemProviderImpl.java | 44 --
.../java/org/apache/doris/fs/GlobListResult.java | 57 --
.../apache/doris/fs/LegacyFileSystemAdapter.java | 221 -------
.../org/apache/doris/fs/LegacyFileSystemApi.java | 146 -----
.../doris/fs/LegacyFileSystemProviderFactory.java | 29 -
.../org/apache/doris/fs/LegacyToNewFsAdapter.java | 92 ---
.../org/apache/doris/fs/LocalDfsFileSystem.java | 199 ------
.../org/apache/doris/fs/PersistentFileSystem.java | 63 --
.../apache/doris/fs/RemoteFileRemoteIterator.java | 51 --
.../main/java/org/apache/doris/fs/RemoteFiles.java | 39 --
.../org/apache/doris/fs/SimpleRemoteIterator.java | 10 +-
.../org/apache/doris/fs/StorageTypeMapper.java | 71 --
.../fs/TransactionScopeCachingDirectoryLister.java | 28 +-
.../org/apache/doris/fs/io/hdfs/HdfsInput.java | 107 ----
.../org/apache/doris/fs/io/hdfs/HdfsInputFile.java | 155 -----
.../apache/doris/fs/io/hdfs/HdfsInputStream.java | 179 ------
.../apache/doris/fs/io/hdfs/HdfsOutputFile.java | 95 ---
.../apache/doris/fs/io/hdfs/HdfsOutputStream.java | 135 ----
.../java/org/apache/doris/fs/obj/S3ObjStorage.java | 38 +-
.../apache/doris/fs/remote/AzureFileSystem.java | 89 ---
.../apache/doris/fs/remote/BrokerFileSystem.java | 712 ---------------------
.../doris/fs/remote/MultipartUploadCapable.java | 53 --
.../org/apache/doris/fs/remote/ObjFileSystem.java | 201 ------
.../apache/doris/fs/remote/RemoteFileSystem.java | 72 ---
.../org/apache/doris/fs/remote/S3FileSystem.java | 150 -----
.../doris/fs/remote/SwitchingFileSystem.java | 131 ----
.../apache/doris/fs/remote/dfs/DFSFileSystem.java | 598 -----------------
.../remote/dfs/DFSFileSystemPhantomReference.java | 44 --
.../apache/doris/fs/remote/dfs/JFSFileSystem.java | 27 -
.../apache/doris/fs/remote/dfs/OFSFileSystem.java | 27 -
.../doris/fs/remote/dfs/OSSHdfsFileSystem.java | 27 -
.../fs/remote/dfs/RemoteFSPhantomManager.java | 127 ----
.../org/apache/doris/persist/gson/GsonUtils.java | 36 --
.../org/apache/doris/backup/BrokerStorageTest.java | 192 ------
.../org/apache/doris/backup/RepositoryTest.java | 12 +-
.../doris/cloud/load/CopyLoadPendingTaskTest.java | 8 +-
.../apache/doris/cloud/stage/StageUtilTest.java | 2 +-
.../apache/doris/datasource/PathVisibleTest.java | 33 +-
.../apache/doris/datasource/hive/HiveAcidTest.java | 176 +++--
.../metastore/PaimonDlfRestCatalogTest.java | 42 --
.../external/iceberg/IcebergHadoopCatalogTest.java | 90 ---
...TransactionScopeCachingDirectoryListerTest.java | 49 +-
.../org/apache/doris/fs/obj/S3FileSystemTest.java | 247 -------
.../doris/fs/remote/RemoteFileSystemTest.java | 195 ------
fe/fe-filesystem/fe-filesystem-azure/pom.xml | 4 +
.../doris/filesystem/azure/AzureFileSystem.java | 2 +-
.../doris/filesystem/azure/AzureObjStorage.java | 7 +-
fe/fe-filesystem/fe-filesystem-broker/pom.xml | 4 +
.../filesystem/broker/BrokerSpiFileSystem.java | 1 +
fe/fe-filesystem/fe-filesystem-cos/pom.xml | 4 +
fe/fe-filesystem/fe-filesystem-hdfs/pom.xml | 4 +
.../doris/filesystem/hdfs/HdfsFileIterator.java | 3 +-
fe/fe-filesystem/fe-filesystem-local/pom.xml | 4 +
.../doris/filesystem/local/LocalFileSystem.java | 3 +-
fe/fe-filesystem/fe-filesystem-obs/pom.xml | 4 +
fe/fe-filesystem/fe-filesystem-oss/pom.xml | 4 +
fe/fe-filesystem/fe-filesystem-s3/pom.xml | 4 +
.../apache/doris/filesystem/s3/S3FileSystem.java | 3 +-
.../apache/doris/filesystem/s3/S3ObjStorage.java | 12 +-
.../org/apache/doris/filesystem/spi/FileEntry.java | 10 +-
.../apache/doris/filesystem/spi/RemoteObject.java | 9 +-
plan-doc/phase4-p48-progress.md | 97 +++
76 files changed, 436 insertions(+), 5103 deletions(-)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemLookup.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemProviderImpl.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/GlobListResult.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/LegacyFileSystemAdapter.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/LegacyFileSystemApi.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/LegacyFileSystemProviderFactory.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/LegacyToNewFsAdapter.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/LocalDfsFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/PersistentFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/RemoteFileRemoteIterator.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/RemoteFiles.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/StorageTypeMapper.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsInput.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsInputFile.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsInputStream.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsOutputFile.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/io/hdfs/HdfsOutputStream.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/AzureFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/BrokerFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/MultipartUploadCapable.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/ObjFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/RemoteFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/S3FileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/SwitchingFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/DFSFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/DFSFileSystemPhantomReference.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/JFSFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/OFSFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/OSSHdfsFileSystem.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/fs/remote/dfs/RemoteFSPhantomManager.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/backup/BrokerStorageTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/external/iceberg/IcebergHadoopCatalogTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/obj/S3FileSystemTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/fs/remote/RemoteFileSystemTest.java
create mode 100644 plan-doc/phase4-p48-progress.md
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
