zghong opened a new pull request, #66595:
URL: https://github.com/apache/doris/pull/66595
### What problem does this PR solve?
Issue Number: close #66527
Problem Summary:
When reading an HTTP resource (e.g. via the HTTP TVF), both BE
(`HttpFileReader`)
and FE (`HttpUtils`) first send an HTTP HEAD request to obtain the file size
from
the `Content-Length` header. Some resources reject HEAD requests -- most
notably
presigned object-storage URLs whose signature covers the HTTP method, so a
URL
signed for GET is rejected with 403 when accessed via HEAD, even though the
same
URL works fine with GET. In that case the HEAD failure aborted opening the
file,
making such presigned URLs unusable.
Fix: a HEAD failure (or a HEAD response without a usable size) no longer
aborts
the open path. Instead it falls back to a ranged GET (`Range: bytes=0-0`),
which
mirrors the actual read path and recovers the total size from the
`Content-Range`
header on a 206 Partial Content response (format `bytes
<start>-<end>/<total>`,
with `*` treated as unknown), or from `Content-Length` on a 200 response.
- BE: `HttpFileReader::open` now issues the HEAD probe without failing on
error;
if no size is obtained it reuses the existing ranged-GET Range-support
probe to
recover the size. A new `HttpClient::get_content_range_total` parses the
total
from the `Content-Range` header.
- FE: `HttpUtils.getHttpFileSize` is split into `tryGetFileSizeWithHead` and
`tryGetFileSizeWithGetRange`, attempting HEAD first and falling back to the
ranged GET, parsing `Content-Range`/`Content-Length` accordingly.
Before the fix presigned GET-only URLs failed to open with a HEAD 403 error;
after the fix the file size is resolved via the ranged GET and the read
succeeds.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test:
`regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy`
- [x] Unit Test: BE `be/test/io/fs/http_file_reader_test.cpp`, FE
`HttpUtilsTest.java`
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
