jensholdgaard commented on issue #24769: URL: https://github.com/apache/datafusion/issues/24769#issuecomment-5462345292
Proposed fix: apache/datafusion#24770. Root cause turned out to be an interaction rather than a pruning bug. `constant_columns_from_stats` folds an all-NULL column to a NULL literal; the predicate then simplifies to a bare constant, at which point `build_pruning_predicates` returns `None` (no column references remain) and `prune_row_groups` falls through with no pruning. So for exactly the files whose statistics carry the most information, the substitution is strictly counterproductive — before it, the pruning predicate's own `col_null_count != row_count` conjunct proved the row group empty. Bisected to #22969 (removing `ListingOptions::collect_stat` in favour of `execution.collect_statistics`, default true) — correct in itself, it just began feeding the substitution on paths that previously had no file statistics. Confirming detail: setting `execution.collect_statistics = false` restores pruning on 55. The PR skips the remaining row groups when the collapsed predicate is `false`/NULL, with a regression test that fails on current main. One correction to the report above: the standalone reproducer uses the DataFrame API, and I noted the SQL path did not reproduce — that is because the SQL literal path did not trigger the substitution in my attempt, not because pruning differs there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
