Gabriel39 opened a new pull request, #69671:
URL: https://github.com/apache/doris/pull/69671
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
An Iceberg partition can guarantee a static row predicate for every row of a
file, but the native scan still reads and evaluates that predicate. For
example, a whole-day timestamp range over a `day(ts)` partition can leave a
filter-only timestamp column in the physical projection.
This change evaluates conservative, exact partition residuals against each
file's actual spec and carries the proven static conjunct identities to
FileScannerV2. Both localized and final scanner filters omit only those
identities. Unconsumed filter-only slots use typed placeholders instead of
physical reads. Predicates and column requirements are rebuilt for each split
and runtime-filter update.
The proof initially covers exact boolean, integer, date and microsecond
timestamp comparisons, IN and null tests. CASTs, compound predicates and other
types retain their row filters. NULL semantics, UTC timestamp carriers,
representable date/time bounds, partition evolution, equality deletes, pending
runtime filters and row-id fetches retain their required checks and columns.
New wire fields are optional; older readers keep their existing filtering
behavior. The connector SPI major and surface baseline are updated together.
Focused assertions cover eliminating two whole-day bounds while retaining a
within-day bound, and reducing a physical projection from two columns to one.
No end-to-end speedup measurement is available.
### Release note
Reduce redundant static filtering and filter-only column reads in Iceberg
FileScannerV2 scans when per-file partition metadata proves the predicates.
### Check List (For Author)
- Test
- [x] Unit Test: 267 Iceberg planner/range/proof tests, 151 connector
SPI tests and 16 FE predicate/wire tests passed. The final 11 proof tests were
rerun after the last change.
- [ ] Regression test: add `test_iceberg_partition_enforced_filters`,
comparing transformed and unpartitioned Spark-written tables across scanners
and batch modes; Groovy compilation passed, cluster execution is pending.
- BE unit tests added for static identities, split reset, runtime
filters, projection reduction, equality deletes and row-id fetches. Local
`run-be-ut.sh` could not link because the third-party installation lacks
`libpaimon_c.a`; execution is pending CI.
- FE Checkstyle, clang-format 16, build hygiene and changed-code
clang-tidy checks passed.
- Behavior changed:
- [x] No. Query results and required residual/deletion semantics are
preserved.
- Does this need documentation?
- [x] No. Internal scan optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
