xudong963 opened a new issue, #23590:
URL: https://github.com/apache/datafusion/issues/23590
### Describe the bug
`EnsureRequirements` can push a sort requirement to the wrong `HashJoinExec`
child when the join has an embedded projection that interleaves columns from
its left and right inputs.
`handle_hash_join` counts projected left-side fields and assumes they occupy
a contiguous prefix of the join output:
```rust
let len_of_left_fields = projected_indices
.iter()
.filter(|ci| ci.side == JoinSide::Left)
.count();
let all_from_right_child = all_indices.iter().all(|i| *i >=
len_of_left_fields);
```
That assumption does not hold for a projection such as `[right.col_a,
left.nullable_col]`. A parent requirement on output index 1
(`left.nullable_col`) is classified as coming from the order-preserving right
child because `1 >= 1`. Its child-local index 0 is then applied to the right
schema, producing a sort on `right.col_a`.
This can produce a physical plan rejected by `SanityCheckPlan`.
### To Reproduce
On `main` at `4b68de61d37839ec40f12d3c1ad8995d5106bbf0`, construct:
```text
SortExec: nullable_col@1 ASC
HashJoinExec:
mode=CollectLeft
join_type=Right
projection=[col_a@2, nullable_col@0]
left schema=[nullable_col, non_nullable_col]
right schema=[col_a, col_b]
```
Run `EnsureRequirements`, followed by `SanityCheckPlan`. The optimized plan
is:
```text
SortPreservingMergeExec: [nullable_col@1 ASC]
HashJoinExec: mode=CollectLeft, join_type=Right, projection=[col_a@2,
nullable_col@0]
DataSourceExec: projection=[nullable_col, non_nullable_col]
SortExec: expr=[col_a@0 ASC], preserve_partitioning=[true]
RepartitionExec: partitioning=RoundRobinBatch(...)
DataSourceExec: projection=[col_a, col_b]
```
`SanityCheckPlan` then reports:
```text
does not satisfy order requirements: [nullable_col@1 ASC].
Child-0 order: [[col_a@0 ASC]]
```
The failure is deterministic. A focused physical-optimizer regression test
reproduces it without SQL or external data.
### Expected behavior
Sort pushdown should determine each required output column's input side from
the projected `ColumnIndex.side` mapping. Because `nullable_col@1` comes from
the non-order-preserving left child, the requirement must not be pushed to the
right child; a sort on `nullable_col@1` should remain above the join.
### Additional context
The same logic is present in DataFusion 54. The correctness issue only
requires an embedded hash-join projection whose left/right fields are reordered
or interleaved; projections that retain a contiguous `[left..., right...]`
layout do not expose it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
