Gabriel39 opened a new pull request, #69614:
URL: https://github.com/apache/doris/pull/69614
### What problem does this PR solve?
Related PR: #69601
Problem Summary:
Elasticsearch HTTP scans always use FileScanner v1, even when
`enable_file_scanner_v2` is enabled. Add an ES TableReader/FileReader pair and
route `FORMAT_ES_HTTP` queries through FileScannerV2 when that session option
is enabled.
The reader exposes a file-local schema, uses MaterializedColumnMapper for
projection and predicate localization, and applies residual filters with v2 row
accounting. It reuses the existing ES query builder, HTTP scroll transport,
authentication, SSL settings, catalog-provided endpoints, and type conversions.
Shared host selection and materialized complex-schema helpers keep the two
scanner paths consistent.
Parse the initial cached response during open so early LIMIT/cancellation
can clear its scroll context. Close the stream on EOF, cancellation, open/read
errors, and split changes, and report close failures once.
The RLE header reach baseline increases by two for the new ES reader and
test translation units, which use the existing table/expr header closure. The
shared parser retains its existing type dispatch, with a documented complexity
suppression on the new projection overload.
### Release note
Support Elasticsearch HTTP queries in FileScannerV2 when
`enable_file_scanner_v2` is enabled. Disabling the option continues to select
FileScanner v1.
### Check List (For Author)
- Test
- [x] Unit Test: 61 ASAN BE tests covering ES v2 projection, doc values,
`_id`, nullable values/arrays, multi-page batching, zero-column row counting,
residual filtering, split advancement, cancellation, error cleanup, and a real
local HTTP scroll server; existing scanner dispatch, legacy ES, and Remote
Doris tests also pass.
- [ ] Regression test: added ES 5/6/7/8 v1/v2 differential coverage with
source/docvalue scans and small batches. Groovy syntax checked; live external
ES regression not run locally.
- [x] clang-format 16, clang-tidy on changed C++ lines, and BE
build-hygiene checks.
- Behavior changed:
- [x] Yes. ES queries honor the v2 session option.
- Does this need documentation?
- [x] No. Uses the existing scanner session option and ES catalog
properties.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
