viirya opened a new pull request, #24336: URL: https://github.com/apache/datafusion/pull/24336
## Which issue does this PR close? - Closes #24335. ## Rationale for this change A `RIGHT`/`FULL` `PiecewiseMergeJoin` with a range predicate drops an unmatched right-side row whose join key is `NULL`. A `NULL` key never matches (`NULL < x` is UNKNOWN), so in a `RIGHT`/`FULL` join the row is unmatched and must still be emitted with NULLs on the left — but `PiecewiseMergeJoinExec` omits it, diverging from `NestedLoopJoin`. ```sql create table l(v int) as values (5); create table r(v int) as values (10), (NULL); select l.v, r.v from l right join r on l.v < r.v; -- drops (NULL, NULL) ``` Root cause: `resolve_classic_join` starts the match scan past the streamed side's `NULL`-keyed rows (they sort to the front under `nulls_first`). Those rows are never revisited, so for `Right`/`Full` they were never added to `unmatched_indices` and got dropped. ## What changes are included in this PR? - In `resolve_classic_join`, when skipping the streamed side's leading `NULL`-key rows, record them as unmatched for `Right`/`Full` joins so they are emitted (with NULLs on the buffered side). ## Are these changes tested? Yes. - Regression test in `pwmj.slt`: a `RIGHT JOIN` over the existing `null_join_*` tables now emits the `(NULL, NULL)` row. The test fails on `main` (the row is dropped) and passes with this change. - Verified more broadly with a differential fuzz against `NestedLoopJoin` (same SQL, `enable_piecewise_merge_join` on vs off): 1200 checks over random `RIGHT JOIN` inputs with `<`/`<=`/`>`/`>=` and high right-side NULL density, 0 mismatches. ## Are there any user-facing changes? `RIGHT`/`FULL` range joins via `PiecewiseMergeJoin` (behind `enable_piecewise_merge_join`, default off) now return unmatched right rows with `NULL` keys, matching `NestedLoopJoin`. No API changes. Note: while investigating I also found a *separate* pre-existing bug — `LEFT`/`FULL` range `PiecewiseMergeJoin` can drop non-matching left rows regardless of nulls. That is out of scope here and I will file it separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
