xudong963 opened a new pull request, #23591: URL: https://github.com/apache/datafusion/pull/23591
## Which issue does this PR close? - Closes #23590. ## Rationale for this change Sort pushdown through a `HashJoinExec` with an embedded projection assumed that all projected left-side columns formed a contiguous prefix. That assumption is not valid when the projection interleaves columns from the two join inputs. For a `CollectLeft` right join projected as `[right.col_a, left.nullable_col]`, a required sort on output column 1 was therefore mistaken for a sort from the right child. The optimizer pushed a sort on `right.col_a` below the join, and `SanityCheckPlan` rejected the resulting plan because it did not satisfy the parent sort on `left.nullable_col`. ## What changes are included in this PR? - Determine whether required sort columns come from the right child by inspecting each projected output column's `JoinSide`, instead of comparing output positions with the number of projected left columns. - Add a regression test with an interleaved hash join projection. The test runs `SanityCheckPlan` and snapshots the valid optimized plan. ## Are these changes tested? Yes. - On unmodified `main`, the new regression test fails in `SanityCheckPlan`: the right input is sorted by `col_a`, while the parent requires `nullable_col`. - With this change, the focused regression test passes. - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption` ## Are there any user-facing changes? There are no API changes. Queries whose hash join projection interleaves columns from both inputs no longer receive an invalid sort pushdown that can fail physical-plan sanity checking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
