viirya opened a new issue, #24335: URL: https://github.com/apache/datafusion/issues/24335
### Describe the bug A `RIGHT`/`FULL` `PiecewiseMergeJoin` with a range predicate drops an unmatched right-side row whose join key is `NULL`. A `NULL` join key never matches under SQL semantics (`NULL < x` is UNKNOWN), so in a `RIGHT`/`FULL` join such a row is unmatched and must still be emitted (with NULLs on the left). `PiecewiseMergeJoinExec` omits it entirely. This is independent of the LeftSemi/LeftAnti existence-join work in #23870 — it is in the classic `RIGHT`/`FULL` path and reproduces on `main`. ### To Reproduce `enable_piecewise_merge_join` on vs off (i.e. `PiecewiseMergeJoin` vs `NestedLoopJoin`) diverge: ```sql set datafusion.optimizer.enable_piecewise_merge_join = true; create table l(v int) as values (5); create table r(v int) as values (10), (NULL); select l.v, r.v from l right join r on l.v < r.v order by 1, 2; ``` PiecewiseMergeJoin returns: ``` +---+----+ | v | v | +---+----+ | 5 | 10 | +---+----+ ``` but the correct result (what `NestedLoopJoin` returns with the flag off) is: ``` +------+------+ | v | v | +------+------+ | 5 | 10 | | NULL | NULL | +------+------+ ``` The right row `v = NULL` is unmatched and should appear as `(NULL, NULL)`. ### Expected behavior `RIGHT`/`FULL` `PiecewiseMergeJoin` emits every unmatched right row, including those with a `NULL` join key, matching `NestedLoopJoin`. ### Additional context Root cause: `resolve_classic_join` in `datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs` starts the match scan past the streamed side's `NULL`-keyed rows (they sort to the front under `nulls_first`). Those rows are never revisited, so for `Right`/`Full` they are dropped instead of being recorded as unmatched. I have a fix + regression test and will open a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
