Shoryamishra61 opened a new pull request, #25647:
URL: https://github.com/apache/datafusion/pull/25647
## Which issue does this PR close?
Closes #25642.
## Rationale for this change
In \OptimizeProjections\, when traversing a \LogicalPlan::Extension\, child
\RequiredIndices\ are constructed using
\RequiredIndices::new_from_indices(necessary_indices)\. Because \
ew_from_indices\ initializes \projection_beneficial\ to \ alse\,
\dd_projection_on_top_if_helpful\ is bypassed during child rewriting in
\map_children\.
While leaf nodes like \TableScan\ prune columns directly against requested
indices, intermediate non-pruning operators like \LogicalPlan::Join\ cannot
eliminate unneeded columns (such as join keys that are needed internally during
matching but not needed downstream by the extension) without an explicit
\Projection\ placed above them. This causes unnecessary columns to flow into
the extension node and prevents downstream physical optimizations like
embedding \HashJoinExec::projection\ via \ProjectionPushdown\.
## What changes are included in this PR?
- In \datafusion/optimizer/src/optimize_projections/mod.rs\, chained
\.with_projection_beneficial()\ when constructing child \RequiredIndices\ from
\xtension.node.necessary_children_exprs(...)\.
- Added unit tests covering:
- Extension node above \Left Join\ (\
est_user_defined_logical_plan_above_join\)
- Extension node with a \Sort\ ancestor above \Left Join\ (\
est_user_defined_logical_plan_above_join_with_sort\)
- Multi-input extension node above \Left Join\ and \TableScan\ (\
est_user_defined_logical_plan_multi_input_above_join\)
## What is the testing strategy for this PR?
- \cargo test -p datafusion-optimizer test_user_defined_logical_plan\ (7
tests pass)
- \cargo test -p datafusion-optimizer optimize_projections\ (61 tests pass)
- \cargo fmt --all -- --check\
- \cargo clippy -p datafusion-optimizer --all-targets -- -D warnings\
- Verified failure before fix via snapshot mismatch and pass after fix.
## Are there any user-facing changes?
No breaking changes or public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
