0AyanamiRei opened a new pull request, #67273:
URL: https://github.com/apache/doris/pull/67273
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Chunked Stream Load requests do not have a `Content-Length`. For
non-streaming formats such as ORC and Parquet, Doris first writes the complete
request body to a local temporary file and then uses a random-access file
reader. However, the BE passed the default `body_bytes` value of `0` to the FE
planner before receiving the body. The generated scan range therefore
advertised the completed local file as a zero-byte file.
Before this change:
- ORC treated the non-empty local file as empty and returned EOF before
parsing it.
- Stream Load completed with `Status: Success`, `NumberTotalRows: 0`, and
`NumberLoadedRows: 0`.
- After preserving the unknown size as `-1`, Parquet still returned zero
rows because its row-group filtering treated `range.size = -1` as having no
readable row groups.
This change:
- Preserves chunked local-file sizes as `-1` when planning, so the local
file reader resolves the actual size after the upload is complete.
- Resolves a Parquet read-to-EOF range to the actual remaining file length
after opening the file, while leaving normal fixed-size split filtering
unchanged.
- Adds end-to-end chunked Stream Load regressions for ORC and Parquet.
After this change, both chunked ORC and chunked Parquet load all 5 rows from
their test files instead of returning a misleading zero-row success.
### Release note
Fix chunked ORC and Parquet Stream Load returning success with zero loaded
rows.
### Check List (For Author)
- Test
- [x] Regression test
- `./run-regression-test.sh --run -d load_p0/stream_load -s
test_stream_load_chunked_orc ...`
- Passed with `NumberTotalRows: 5`, `NumberLoadedRows: 5`, and
`LoadBytes: 1262`.
- `./run-regression-test.sh --run -d load_p0/stream_load -s
test_stream_load_chunked_parquet ...`
- Passed with `NumberTotalRows: 5`, `NumberLoadedRows: 5`, and
`LoadBytes: 1402`.
- `./run-regression-test.sh --run -d load_p0/stream_load -s
test_load_with_decimal ...`
- Passed for the existing fixed-length Parquet, ORC, CSV, and JSON
paths.
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- `BUILD_TYPE=ASAN ./build.sh --be -j48`
- `build-support/run-clang-tidy.sh` passed with no warnings on
changed lines.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Chunked ORC and Parquet uploads now read the completed local
file and load its rows instead of being treated as empty.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
