Gabriel39 opened a new pull request, #68596:
URL: https://github.com/apache/doris/pull/68596
### What problem does this PR solve?
Arrow Flight SQL can return a string array containing invalid UTF-8, for
example from `SELECT unhex('84')`. Arrow string builders accept the bytes, but
clients such as PyArrow fail when decoding the result.
Validate each completed array in `ArrowFlightArrowBlockConvertor` before
publishing the batch. Reuse Arrow's recursive validation to cover nested
strings and per-value UTF-8 boundaries. Return an invalid-argument error
identifying the column ordinal, field name and Arrow validation failure. Binary
values retain their byte representation, and NULL payloads are handled through
the serialized validity bitmap.
Add six BE unit tests and a Flight SQL regression suite covering malformed
encodings, separate rows whose concatenation is valid UTF-8, subsequent
batches, nested arrays/structs/map keys and values, large strings, valid
Unicode, NULLs and binary results.
### Validation
- Reproduced four failing tests before the fix; the two valid-data/NULL
tests passed.
- ASAN: all 46 tests in `ArrowBlockConvertorTest` and
`DataTypeSerDeArrowTest` passed after the fix.
- clang-format 16 passed for all three affected C++ files.
- The new Groovy regression suite compiled successfully. Live Flight SQL
regression execution remains pending CI.
The local BE unit-test build selected the relevant suites and their test
support files; the build configuration was restored and is not part of this PR.
### Release note
Arrow Flight SQL now rejects invalid UTF-8 string results with a descriptive
server-side error instead of returning malformed Arrow data.
### Check List (For Author)
- Test
- [x] Regression test added (live execution pending CI)
- [x] Unit Test
- Behavior changed:
- [x] Yes. Invalid Arrow Flight string payloads fail on the server
before the affected batch is returned.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
