mohitgurav20 opened a new pull request, #25811: URL: https://github.com/apache/datafusion/pull/25811
Closes #25797 Rationale for this change Parquet Bloom filter pruning currently loads filters for all relevant predicate columns before evaluating whether a row group can be excluded. For queries with multiple filtering conditions, this results in unnecessary I/O reads if an earlier filter already proves the group cannot match. By incrementally evaluating Bloom filters and stopping reading additional filters for a row group as soon as a condition proves it cannot match, we can skip unnecessary reads and release memory eagerly. This brings a solid performance bump for observability workloads filtering across several columns over remote storage. What changes are included in this PR? Incremental Evaluation: Folded the Bloom filter evaluation directly into the load_bloom_filters async loop to evaluate immediately as each column is fetched. Short-circuiting: As soon as predicate.prune(&row_group_filters) rules out a row group, it breaks out of the loop and skips reading any remaining columns for that group. State Machine Optimization: Dropped the BloomFiltersLoadedParquetOpen and PruneWithBloomFilters states entirely. Because we evaluate eagerly during I/O, we no longer need to retain all of those filters in memory for a secondary pass. The state machine now transitions cleaner directly to BuildStream. Access Plan Updates: Added a mutable accessor access_plan_mut to RowGroupAccessPlanFilter to support skipping row groups directly from within the loader. Are these changes tested? Yes, this relies on existing Parquet pruning test suites which will validate that correctness is maintained while avoiding unnecessary reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
