morningman opened a new pull request, #66589:
URL: https://github.com/apache/doris/pull/66589
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66403, #66399
Problem Summary:
#66403 added a per-range table reader rebuild to `FileScannerV2`: whenever a
range carries a table format different from the one `_table_reader` was built
for, the reader is thrown away and rebuilt. This removes it again, because
nothing produces the situation it handles.
It was added ahead of a connector that reads a table as a lake plus the log
written after it — fluss, proposed in #66399. That connector planned its lake
half through a sibling connector and its own half itself, so one scan node was
handed ranges of two table formats; a reader built once from the first range is
then given a foreign one and does not fail cleanly, it fails as whatever that
reader makes of it. Since which ranges share a scanner is the engine's
assignment, the same query would succeed or fail by how the ranges happened to
be dealt out.
That connector has since been changed — on review feedback — to carry the
one table format on every range of a scan and to make the per-range choice
inside its own table reader, which is the arrangement paimon and hudi already
use (`PaimonHybridReader`, `HudiHybridReader`). So the producer this mechanism
was built for no longer exists, in this tree or in that branch.
**Nothing else produces a mixed-format scan node.** Checked, rather than
assumed:
- every `ConnectorScanRange` implementation in the tree returns a
**constant** table format string — `paimon`, `hudi`, `iceberg`, `jdbc`,
`max_compute`, `trino_connector`, `es`, `adbc`, `trino` — with one exception;
- the exception is `HiveScanRange`, which returns `transactional_hive` for
ACID partitions and `hive` for the rest, so a hive ACID table genuinely can
plan two format strings into one node;
- but `transactional_hive` appears in none of
`FileScannerV2::is_supported`'s whitelists (`is_supported_table_format`,
`is_supported_arrow_table_format`, `is_supported_jni_table_format`), so such a
range never enters this scanner at all — it stays on the V1 path.
Removing the mechanism also removes a subtlety it carried. It rebuilt the
reader but deliberately **not** the expression contexts, because
`_init_expr_ctxes` is not idempotent. That is correct only while every reader a
scanner could switch between annotates its projected columns identically — a
property each newly added table format had to preserve by accident, with
nothing checking it. With one reader per scanner, that property is structural
instead.
Net effect: `-95` lines, no behaviour change.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
The unit test removed here is the one that covered the removed method; it
drove `_rebuild_table_reader_if_format_changed` directly and has nothing left
to call. No other test in `file_scanner_v2_test.cpp` references the method or
the `_table_reader_format` member — the shared `range_with_format` helper it
used stays, with nine other callers.
Verified locally: `be/src/exec/scan/file_scanner_v2.cpp` and
`be/test/exec/scan/file_scanner_v2_test.cpp` compile clean under the project's
own flags (`-Wall -Wextra -Werror`) with the mechanism gone, and `clang-format
--dry-run -Werror` is clean on all three files. The identical deletion has been
running in the #66399 branch, where `FileScannerV2*:FileScannerTest*:Fluss*`
(73 tests) and the fluss end-to-end suites (12 suites, 0 skipped) are green
against a real fluss + flink + paimon cluster.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
