alamb opened a new issue, #26542: URL: https://github.com/apache/datafusion/issues/26542
### Is your feature request related to a problem or challenge? `EliminateFilter` simplifies filter predicates using filter semantics (a `NULL` result rejects a row exactly like `FALSE`), but it only walks positive `AND` / `OR` trees and treats every other node as an opaque leaf (added in #24258). `InList` is therefore never simplified, so a `NULL` literal inside an `IN` list survives into the physical plan when the list is kept as an `IN` list: ```sql -- on main, lists with 4+ members are kept as IN lists explain select * from t where x in (1, 2, 3, NULL); -- Filter: t.x IN ([Int32(1), Int32(2), Int32(3), Int32(NULL)]) ``` Today, lists with 1-3 members are rewritten to `x = 1 OR x = 2 OR x = NULL`, the simplifier folds `x = NULL` to `NULL`, and `EliminateFilter` drops it, so the short-list case ends up as `x = 1 OR x = 2`. #25187 proposes keeping short lists as `IN` lists for some types, which would make them behave like the longer lists above (see the change to `eliminate_outer_join.slt` in that PR). This matters for Parquet pruning. The per-value pruning expansion in `datafusion/pruning/src/pruning_predicate.rs` turns the `NULL` member into a `x_min <= NULL AND NULL <= x_max` disjunct. That evaluates to `NULL`, `false OR NULL` is `NULL`, and the pruner treats `NULL` as "may match", so a predicate like `x IN (1, 2, 3, NULL)` never prunes a row group or page. Bloom filter pruning degrades the same way: a `NULL` scalar in the `LiteralGuarantee` hits the "unsupported type, assume present" branch in `bloom_filter.rs`. The existing expected output at `push_down_filter_parquet.slt` for `IN (SET) ([11, NULL])` shows the resulting shape. #24711 observed the same "degrades to always keep" behavior for the compact pruning path and fixed it only for lists over 20 members. ### Describe the solution you'd like Extend `simplify_filter_predicate` in `datafusion/optimizer/src/eliminate_filter.rs` with an `Expr::InList` arm that applies filter semantics: - **Positive `IN` with some null literal members**: drop the null literals. `x IN (1, 2, NULL)` becomes `x IN (1, 2)`. The predicate is `TRUE` for exactly the same rows, since a `NULL` member can only turn `FALSE` into `NULL`, and a filter rejects both. - **Positive `IN` with only null literal members**: never `TRUE`, so classify as rejects-all (`EmptyRelation`). - **`NOT IN` with any null literal member**: never `TRUE`, so classify as rejects-all. Note this must *not* drop the member: `x NOT IN (1, NULL)` is `FALSE` for `x = 1` and `NULL` otherwise. - Only typed null *literals* count. A nullable column or expression in the list is a runtime matter and must stay. - `is_safe_to_discard` should learn `InList` over columns and literals so the rejects-all cases can discard the expression. This is only sound inside `EliminateFilter`. It must not be done in `ExprSimplifier` or the `IN` list shortening rewrite, which run in every context: `SELECT 3 IN (1, NULL)` must stay `NULL`, and `WHERE NOT (x IN (1, NULL))` must keep rejecting `x = 3`. That is the same reason #8688 only implemented the `NULL IN (...)` half. With this in place the pruning problem disappears for filters, which is the only place pruning applies, for lists of every length. ### Describe alternatives you've considered - Have the short-list retention in #25187 opt out when a list contains a null literal. Restores parity with `main` for 1-3 member lists but leaves longer lists unpruned. - Drop null members from positive `IN` lists inside the per-value pruning expansion in `pruning_predicate.rs`, as the compact path already does. Sound because pruning asks only whether the predicate can be `TRUE`, but it only fixes pruning, not the executed filter, and #24711 notes the interaction with `identify_fully_matched_row_groups`. ### Additional context - #8688: original request for `NULL` in-list simplification; the `x IN (NULL, ...)` half was dropped as unsound in general expression context. - #24258: introduced the filter-semantics simplification that this would extend. - #24711: same pruning degradation, fixed only for the compact path (lists over 20 members). - #25187: makes 2-3 member lists on several types hit this path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
