Gabriel39 opened a new pull request, #69114:
URL: https://github.com/apache/doris/pull/69114
### What problem does this PR solve?
`hive_on_hms_and_dlf` fails while inserting into a partitioned Hive table
whose location uses the supported legacy OSS form
`oss://<bucket>.<endpoint>/<path>`. For example:
```text
oss://sample-bucket.oss-cn-hangzhou.aliyuncs.com/warehouse/table/part=value/
```
The native OSS filesystem parses the entire authority as the bucket and
passes `sample-bucket.oss-cn-hangzhou.aliyuncs.com` to the SDK. The request
fails with `The bucket name "..." is invalid` before it can access the intended
object. The failure occurs in the HMS/OSS portion of the suite; it is not a
timestamp result mismatch or a DLF credential failure.
FE-core already normalizes this legacy form for paths sent to BE, but FE
filesystem operations can receive the original location retained in HMS. The
native OSS provider therefore also needs the legacy bucket extraction. After
this change, the example above produces bucket `sample-bucket` and key
`warehouse/table/part=value/`, using the configured OSS endpoint.
Centralize the correction in the OSS provider and use it for all URI-based
object requests. Let the shared filesystem delegate URI parsing to that
provider as well: bulk/recursive deletion extracts buckets before calling the
SDK and must follow the same rule. Preserve encoded keys and literal `+`
characters; leave HTTP(S) parsing and other providers unchanged, including
valid dotted S3 bucket names.
### Release note
Fix Hive operations against legacy OSS locations containing a bucket and
endpoint in the URI authority.
### Check List (For Author)
- Test:
- [x] Unit Test: added seven focused tests. The initial list, HEAD and
bulk-delete regressions failed on the original implementation and passed after
the fix.
- [x] `cd fe && mvn -pl fe-filesystem/fe-filesystem-oss -am test
-Dmaven.build.cache.enabled=false`: 356 tests passed, including 50 OSS and 49
filesystem SPI tests. Maven Checkstyle passed with zero violations.
- Coverage includes OSS/S3/S3A aliases under the OSS provider, both
addressing settings, encoded partition keys, custom endpoints and bucket roots,
write/copy/delete, bulk/recursive deletion, path-style HTTP, unsupported
schemes and dotted S3 buckets.
- `run-fe-ut.sh --run OssObjStorageTest` was attempted but stopped on a
local Thrift compiler mismatch (installed 0.16.0, required 0.24.0). The focused
modules above do not require Thrift generation. The full cloud-backed Hive
regression has not been rerun locally.
- Behavior changed:
- [x] Yes: native OSS requests accept legacy bucket.endpoint locations
consistently with existing FE path normalization.
- Does this need documentation?
- [x] No: restores existing URI compatibility without adding configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
