mrhard9090 opened a new pull request, #25772:
URL: https://github.com/apache/datafusion/pull/25772
## Which issue does this PR close?
- Closes #25704.
## Rationale for this change
A query that joins on `IS NOT DISTINCT FROM` and then inner joins another
table on different keys loses the rows whose keys are NULL:
```sql
SELECT t1.a, t1.b, t2.a, t3.b
FROM t1 JOIN t2 ON t1.a IS NOT DISTINCT FROM t2.a
JOIN t3 ON t1.b = t3.b
```
`EliminateCrossJoin` flattens every inner join under the root join, pools
their keys, and rebuilds the joins with the root join's `null_equality`. The
nested `NullEqualsNull` join is rebuilt as `NullEqualsNothing`, so NULL keys
stop matching. The reverse case is wrong too: a nested `NullEqualsNothing` join
under a `NullEqualsNull` root would start matching NULLs.
## What changes are included in this PR?
`flatten_join_inputs` takes the `null_equality` that the joins will be
rebuilt with. It flattens an inner join only when that join has no keys (a
cross join) or its `null_equality` matches. Any other inner join stays an input
as a whole, and the rule optimizes it on its own when it reaches it through
`rewrite_children`. For the plain `Join` root, `null_equality` is now set
before flattening, not after.
## What is the testing strategy for this PR?
- `keep_nested_join_with_other_null_equality` in `eliminate_cross_join.rs`
builds `t1 JOIN t2` with `NullEqualsNull` under a `NullEqualsNothing` join with
`t3`. It checks that after the rule, the `t1.a = t2.a` join still has
`NullEqualsNull`. It fails without the change, when that join comes back as
`NullEqualsNothing`.
- `join_is_not_distinct_from.slt` gets the query from the issue. It now
returns both rows, including the NULL-key row.
- `cargo test -p datafusion-optimizer --lib eliminate_cross_join` passes (27
tests). The same goes for the `join_is_not_distinct_from` sqllogictest, `cargo
fmt --all` and `cargo clippy -p datafusion-optimizer --no-deps --all-targets
--all-features -- -D warnings`. Without `--no-deps`, clippy stops earlier on an
`unnecessary_semicolon` in `datafusion-common` (`rounding.rs`) on Rust 1.98.1,
which this PR doesn't touch.
## Are there any user-facing changes?
Queries like the one above return the rows with NULL keys again. There are
no API changes.
This PR was written with help from Claude Opus 5.5. I checked the change
against the rule's flattening and rebuild logic and ran the tests above myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
