rgbuilds opened a new pull request, #26441: URL: https://github.com/apache/datafusion/pull/26441
## Which issue does this PR close? Closes #26130. ## Rationale for this change When files contain disjoint ranges of a filtered column, ListingTable groups all files before file-statistics pruning. The matching files can therefore land in a single scan partition, and scan statistics still include files that cannot match. In the eight-file example from the issue, `d >= 20 AND d < 40` currently places both matching files in the second of four groups and estimates 80 rows. Pruning before grouping retains only `f2` and `f3`, puts them in separate groups, and aggregates 20 survivor rows. ## What changes are included in this PR? - Pass non-partition filters through a private listing path and reuse `PruningPredicateBuilder` with already collected file statistics before splitting regular scan file groups. - Keep files with missing/inexact statistics or evaluation errors, retain normal row filtering, and avoid evaluating volatile filters during planning. - Bypass the optimization for custom expression adapters and declared output partitioning, preserving partition identities and empty trailing groups. - Recompute aggregate statistics from survivors without additional object-store requests. - Add regression coverage and update physical-plan/runtime-metric expectations for files eliminated during planning. No public API change. ## What is the testing strategy for this PR? On the investigated baseline (`a4b8198cf`), local validation passed 12,579 distinct Rust tests and 533 SQL logic files. This includes 23 listing unit tests, 10 focused pruning integration tests, all 246 Parquet integration tests, and 126 extended fuzz tests. The no-extra-I/O test compares object-store requests during planning. Coverage includes the eight-file grouping example, partially matching files, missing and inexact bounds, NULL semantics, Hash/Range partition layouts, Hive/data predicates, LIMIT/OFFSET, custom adapters, volatile UDF invocation counts, survivor statistics, and conservative filtered-statistics precision. Nine SQL snapshot files change physical-plan expectations only. On current upstream main (`5f2158269`), all 1,459 core integration tests (including all 10 pruning regressions and the no-extra-I/O test), all 247 Parquet tests, and all 535 SQL logic files pass. The full extended workspace/fuzz run is still in progress; its final result will be recorded separately. Formatting, diff whitespace, TOML formatting, and license headers on all 2,331 tracked project source files pass. The earlier full workspace Clippy run passed with explicit allowances for pre-existing upstream/toolchain diagnostics. The mandatory strict Clippy invocation also fails on the updated main branch with Rust 1.99, on three pre-existing `clippy::mutable_key_type` diagnostics in untouched `datafusion/optimizer/src/push_down_filter.rs`. The CI-style Clippy invocation (`--workspace --all-targets --features avro,integration-tests,extended_tests -- -D warnings`) passes. The full lint wrapper stops at its license walk because locally exported third-party SQLite fixture snapshots lack project headers; the separate tracked-project-source license check passes. Subsequent wrapper steps have not been run. No lint suppression or unrelated optimizer change is included in this PR. No performance speedup is claimed: the prior Hive-only planning benchmark is not representative of data-column pruning and ran under differing concurrent load. ## Are there any user-facing changes? With statistics collection enabled, regular listing scans can contain fewer files and report tighter statistics in EXPLAIN. Query results and row filtering are unchanged. Unsupported or inconclusive statistics keep files, and declared partition layouts are preserved. This contribution was developed with AI assistance. The design and safeguards are described above; performance under distributed execution remains unmeasured. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
