alamb opened a new pull request, #24395: URL: https://github.com/apache/datafusion/pull/24395
## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/24393 - Related to https://github.com/apache/datafusion/issues/20324 and https://github.com/apache/datafusion/issues/3463 **Note: this is a prototype and is NOT READY FOR REVIEW.** It exists to evaluate whether decoupling the I/O pattern from filter pushdown avoids the I/O-related regressions that have blocked enabling `pushdown_filters` by default. ## Rationale for this change Enabling `pushdown_filters` today also changes the I/O pattern: instead of one request per row group, the reader sequentially fetches the columns for each filter predicate, then (after evaluating the filters) the remaining projected columns. When a file has no offset index (e.g. the ClickBench dataset), those extra round trips cannot prune any bytes, so they are pure overhead. This PR separates "filter evaluation mechanics" from the I/O pattern via a new config option, `datafusion.execution.parquet.progressive_io`: | `pushdown_filters` | `progressive_io` | behavior | |---|---|---| | false | (any) | unchanged: one request per row group | | true | **false (default)** | filters evaluated during decode, but all column chunks needed by the filters or the projection are fetched with **one request per row group** | | true | true | progressive fetch, but only when the file has an offset index (where it can actually prune bytes); otherwise one request per row group | ## What changes are included in this PR? - New `progressive_io` config option (config, proto, docs, `ParquetSource::with_progressive_io`), resolved table-or-session like `pushdown_filters` - `RowFilterGenerator` now exposes the union of the filter predicates' `ProjectionMask`s - In the opener, when one-shot I/O applies, the byte ranges of all column chunks in (filter mask ∪ projection mask) are precomputed per row group; `PushDecoderStreamState::transition` expands the first `NeedsData` request of each row group to those ranges, so the whole row group is fetched in one `get_byte_ranges` call. The decoder's buffered-range containment check then satisfies all subsequent phases with zero additional I/O. No arrow-rs changes were needed. Known prototype limitations: - When page-index pruning produces a row selection that actually prunes rows, the scan falls back to progressive I/O (one-shot whole-chunk ranges would defeat the page pruning). A future refinement could fetch selection-pruned page ranges in a single request instead. - One-shot ranges are whole column chunks, so peak buffered bytes per row group equal the compressed size of all needed chunks (same as the non-pushdown reader today). ## Are these changes tested? Yes: - New object-store request-count tests in `object_store_access.rs` show that with `pushdown_filters=true` and the default `progressive_io=false`, the scan issues exactly one data request per row group with byte ranges identical to the non-pushdown scan, and that `progressive_io=true` restores the current progressive pattern - Existing filter pushdown tests and sqllogictests pass unchanged ## Are there any user-facing changes? Yes: a new configuration option, and (intentionally) a changed default I/O pattern for users who enable `pushdown_filters`. Benchmarks (`clickbench_pushdown`) to follow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
