zhuqi-lucas opened a new pull request, #23696: URL: https://github.com/apache/datafusion/pull/23696
## Which issue does this PR close? - Closes #23067. ## Rationale for this change Builds on the `peek_next_row_group` API landed in [apache/arrow-rs#10158](https://github.com/apache/arrow-rs/pull/10158) (in released `arrow-rs 59.1.0`). When Parquet stats prove that every row of a row group already satisfies the pushdown predicate (`fully_matched`), running the per-row `RowFilter` inside that row group is pure overhead — every row passes anyway. This change installs an *empty* `RowFilter` on fully-matched runs and only pays the row-level machinery on RGs that still need filtering. Two-phase reads for those RGs also skip fetching the filter columns when they aren't in the projection — this pattern is common on time-partitioned tables (`WHERE ts >= X AND ts < Y` where the RG's min/max is contained in `[X, Y)`) and on the middle stretch of a `TopK` scan where the dynamic threshold is still loose. ## What changes are included in this PR? - `access_plan.rs` — `ParquetAccessPlan` now tracks per-RG `fully_matched` state, produced during static pruning. - `metrics.rs` — new `row_filter_skipped_fully_matched` counter to observe how often the toggle fires. - `push_decoder.rs` — `RgPlanEntry` carries the per-RG toggle; `RowFilterContext` swaps between the real filter (from the `prebuild_row_filter_candidates` cache) and an empty filter as the decoder crosses row-group boundaries via `peek_next_row_group()`. - `row_filter.rs` — split `build_row_filter` into `prebuild_row_filter_candidates` (once per file: tree walk + candidate construction) and `row_filter_from_prebuilt` (per RG boundary: cheap bind against the fresh array readers). The existing public `build_row_filter` API is preserved for callers that don't need per-RG toggling. - `opener/mod.rs` — wires the prebuilt cache into the decoder builder; skips page-index loading for fully-matched RGs (their per-row filter is a no-op, so page pruning saves nothing). - `sort.rs` — sort-order-aware RG reorder preserves the per-RG toggle through reordering. ## Are these changes tested? Yes. - `fully_matched_rgs_skip_row_filter` (`datafusion/core/tests/parquet/dynamic_row_group_pruning.rs`) — 4 RGs of 3 rows each, predicate `v >= 3` makes RG 0 a straddler and RGs 1..=3 fully matched. Asserts (a) results are correct and (b) `row_filter_skipped_fully_matched` fires at least once at the RG 0 → RG 1 boundary. - All 6 pre-existing `dynamic_row_group_pruning` integration tests still pass. - Full `datafusion-datasource-parquet` lib suite (158 tests) passes. Expected benchmark impact (I'll follow up with numbers): - **`sort_tpch`** (target workload): the tightening dynamic threshold produces fully-matched runs on the way up, then fully-pruned RGs after the threshold snaps — expect additional 10–30% on the queries already accelerated by #22450. - **TPCH Q6 / Q12 / Q14** (date-range filters): a portion of RGs are fully in-range; expect ~5–15%. - **ClickBench**: near-zero — the `URL LIKE …` / equality filters used there rarely fully cover an RG. ## Are there any user-facing changes? - New `row_filter_skipped_fully_matched` metric appears in `EXPLAIN ANALYZE` on parquet scans (0 when the toggle didn't fire). - No public API changes; `build_row_filter` still exists and behaves identically for non-toggling callers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
