jayzhan211 opened a new pull request, #25489:
URL: https://github.com/apache/datafusion/pull/25489
## Which issue does this PR close?
- No separate issue. Split out of #25217 so it can be reviewed on its own.
## Rationale for this change
A `SortMergeJoinExec` with a join filter returns wrong rows when the filter's
`column_indices` do not list every left column before every right column.
`JoinFilter::swap` produces exactly that layout, so any plan that goes
through
the public `SortMergeJoinExec::swap_inputs()` is affected, as is any
`JoinFilter` built by hand or by a custom optimizer rule. For example, a Left
join on `t2.b1 = t1.b1 AND t2.a2 > t1.a1` whose filter indices are
`[Right(a1), Left(a2)]` is evaluated as `a1 > a2`:
expected actual
| 10 | 4 | 1 | 4 | 7 | | 10 | 4 | | | |
| 20 | 5 | | | | | 20 | 5 | 21 | 5 | 8 |
When the misplaced columns share a type there is no error, only wrong
results.
Queries planned from SQL are not affected today: the physical planner always
builds the filter with left columns first, and `JoinSelection` does not swap
sort-merge joins.
## What changes are included in this PR?
`get_filter_columns` (used by the materializing SMJ stream) collected all
left
columns, then all right columns, and the result was zipped against the
filter's
intermediate schema, which is in `column_indices` order. It now walks
`column_indices` once and takes each column from the side it names.
This matches what the semi/anti/mark stream already does in
`bitwise_stream.rs`. Columns with `JoinSide::None` are skipped, as before.
## What is the testing strategy for this PR?
New unit test `join_left_with_filter_columns_right_before_left` in
`sort_merge_join/tests.rs`: a Left join whose filter lists a right column
before a left one. It fails on `main` with the wrong rows shown above and
passes with the fix. The existing `sort_merge_join` tests pass unchanged.
## Are there any user-facing changes?
No API changes. Sort-merge joins with a filter in right-before-left column
order now return correct results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
