jensholdgaard commented on PR #24770: URL: https://github.com/apache/datafusion/pull/24770#issuecomment-5468266777
The `cargo test (amd64)` failure was real and pointed at a scoping problem in my first push — fixed in the update. The nine failing `evolved_schema*` / `test_pushdown_with_missing_*` tests exercise the *other* path that substitutes NULL for a column: files missing the column entirely (schema evolution). My original check keyed only on "the simplified predicate is a constant NULL/false literal", so it also fired there, pruning row groups those tests expect to see in the raw scan output and shifting their pushdown metric counts. Semantically that pruning would be sound too (a file missing `c2` cannot satisfy `c2 = 2`), but it is a behaviour change well beyond the regression this PR fixes, and partition-value-driven collapse is similarly already `FilePruner`'s territory. The update narrows the skip with a `stats_constants_in_predicate` guard: the collapse only counts as statistics-proven when the original predicate referenced a column that `constant_columns_from_stats` proved constant for this file. With that: - the nine previously failing tests pass again unchanged, - the new regression test still fails without the fix and passes with it, - `cargo test -p datafusion-datasource-parquet --lib opener::` — 47 passed, - `cargo test -p datafusion --lib datasource::physical_plan::parquet::tests` — 35 passed (the 4 `parquet_exec_with_*` failures on my machine reproduce on an untouched checkout and pass in CI, so they are environmental). If maintainers would rather generalise the skip to the missing-column collapse as well, I'm happy to do that in a follow-up with the corresponding test updates — it just seemed wrong to smuggle a second behaviour change into a targeted regression fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
