[
https://issues.apache.org/jira/browse/HDFS-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18066021#comment-18066021
]
ASF GitHub Bot commented on HDFS-11161:
---------------------------------------
yangdong2398 opened a new pull request, #8347:
URL: https://github.com/apache/hadoop/pull/8347
### Description of PR
JIRA: https://issues.apache.org/jira/browse/HDFS-11161
Add `hadoop-bos` module to `hadoop-cloud-storage-project` providing Baidu
Object Storage (BOS) FileSystem implementation using the `bos://` URI scheme.
**Key features:**
- `BaiduBosFileSystem`: core Hadoop FileSystem implementation for BOS
- Support for both flat and hierarchical namespace modes
- Multipart upload for large files
- CRC32C checksum compatible with HDFS for DistCp
- Pluggable credential providers (configuration-based and
environment-variable-based)
- Hadoop FileSystem contract tests
- Shaded BOS SDK and transitive dependencies (jackson, httpcomponents,
guava, commons-logging, commons-lang, commons-codec) under `bfs.*` prefix to
avoid classpath conflicts
**Build integration:**
- Integrated into `hadoop-project/pom.xml` (dependency management),
`hadoop-cloud-storage-project/pom.xml` (module), `hadoop-cloud-storage/pom.xml`
(umbrella dependency), `hadoop-cloud-storage-dist/pom.xml` (dist packaging)
- Standard `core-site.xml` + XInclude `auth-keys.xml` test credential pattern
- `tests-off`/`tests-on` profiles to auto-skip tests when credentials are
absent
### How was this patch tested?
- Unit test: `TestConfigurationCredentialsProvider` validates credential
provider loading from Hadoop configuration
- Contract tests: Full set of Hadoop FileSystem contract tests
(`TestBosContractCreate`, `TestBosContractDelete`, `TestBosContractRename`,
`TestBosContractSeek`, etc.) — require valid BOS credentials in `auth-keys.xml`
- CI: Apache Yetus pre-commit checks pass (compile, javadoc JDK 17/21,
SpotBugs, checkstyle, ASF license, whitespace)
### Dependencies and Licensing
All new dependencies (Baidu BOS SDK) and transitive dependencies are
properly shaded under `bfs.*` prefix. BOS SDK is under Apache License 2.0,
compatible with ASF 2.0 license requirements. No changes required to LICENSE,
LICENSE-binary, or NOTICE-binary files as dependencies are shaded and not
exposed in distribution.
> Incorporate Baidu Yun BOS file system implementation
> ----------------------------------------------------
>
> Key: HDFS-11161
> URL: https://issues.apache.org/jira/browse/HDFS-11161
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Reporter: Faen Zhang
> Priority: Major
> Labels: pull-request-available
> Original Estimate: 840h
> Remaining Estimate: 840h
>
> Baidu Yun ( https://cloud.baidu.com/ ) is one of top tier cloud computing
> provider. Baidu Yun BOS is widely used among China's cloud users, but
> currently it is not easy to access data laid on BOS storage from user's
> Hadoop/Spark application, because of no original support for BOS in Hadoop.
> This work aims to integrate Baidu Yun BOS with Hadoop. By simple
> configuration, Spark/Hadoop applications can read/write data from BOS without
> any code change. Narrowing the gap between user's APP and data storage, like
> what have been done for S3 and Aliyun OSS in Hadoop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
