asg0451 opened a new pull request, #26438: URL: https://github.com/apache/datafusion/pull/26438
🤖 *Written by Claude (an AI assistant) and posted from Miles's account.* ## Which issue does this PR close? - Part of #21554. ## Rationale for this change A Parquet scan with a predicate builds a `PruningPredicate` for each file's `FilePruner` when it opens the file, and again whenever a dynamic filter in the predicate changes. A build snapshots the dynamic filters, simplifies the predicate and rewrites it in terms of statistics, which costs much more than evaluating the result against one file's statistics. Files with the same partition values build the same pruning predicate, so a scan of many small files does that work once a file. It shows most under a `TopK` over many files that carry statistics: the first files fill the TopK, and its dynamic filter then prunes most of the rest at open, each after building the pruning predicate from scratch. In a service of ours whose query reads ~350 `PartitionedFile`s (row ranges of Parquet files, each with min/max statistics for the sort column, in one partition) under a newest-first `TopK`, `FilePruner::should_prune` was 45% of a query's CPU: 31% building the pruning predicate, 13% evaluating it. ## What changes are included in this PR? - `PruningPredicateCache` in `datafusion-pruning`: one slot holding the last pruning predicate built, with the predicate it was built from (after partition values are replaced with literals), the file schema, and the predicate's `snapshot_generation`. Clones share the slot. - `FilePruner::with_cache`, additive: `should_prune` takes the pruning predicate from the cache, and builds it only when the predicate, the schema or the generation differs. - `ParquetMorselizer` keeps one cache per partition and gives it to each file's pruner. For review: - The generation is read before the build, so a cached pruning predicate was built from that generation of the dynamic filters or a later one. A later dynamic filter is as correct to prune with; it's what a file opened a moment later would get. - One slot, not a map: files with the same partition values are usually next to each other in a partition's file group, and one slot bounds the memory at one pruning predicate a partition. Files whose partition values alternate build each time, as now. - The slot's lock is held while building, so two files of a partition don't build the same one at once. - `predicate_creation_errors` now counts a failed build once, not once for each file that would have built it again. - #21566 caches the row-group pruning setup for static predicates, and skips dynamic filters and literal replacement. This covers the `FilePruner`, dynamic filters included, so the two don't overlap. ## What is the testing strategy for this PR? - `file_pruner::tests::files_with_the_same_predicate_share_its_pruning_predicate`: under a `DynamicFilterPhysicalExpr`, a second file reuses the first's pruning predicate; after `update`, the next check builds a new one, which the files after it reuse; each file is pruned or kept as before. - `file_pruner::tests::files_with_different_predicates_build_their_own`. - `opener::test::test_prune_on_partition_values_of_each_file_of_a_scan`: one morselizer opens files with different partition values, interleaved, under a dynamic filter, and each is pruned or kept by its own partition value. Benchmark: `parquet_pruning_setup_cache/same_schema_files` (128 files, static predicate), main (5f2158269) against this branch, four runs each taking turns on a laptop, LTO off: median 11.1 ms → 10.1 ms (−9%), with about ±10% between runs. Not known: - A local TopK variant of that bench (`SELECT id FROM t ORDER BY id LIMIT 10` over the same files) was within noise. I haven't checked whether its files reach the `FilePruner`, or are skipped before it. ## Are there any user-facing changes? `PruningPredicateCache` and `FilePruner::with_cache` are new public API in `datafusion-pruning`. Nothing breaks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
