haohuaijin opened a new pull request, #21527:
URL: https://github.com/apache/datafusion/pull/21527
## Which issue does this PR close?
- Closes #21526.
## Rationale for this change
`FilterPushdown` does not push filters through `SortExec` because `SortExec`
lacks an implementation of `handle_child_pushdown_result`. When a `FilterExec`
sits above a plain `SortExec` (no fetch), the filter can safely be moved below
the sort without changing semantics, since sorting preserves all rows.
## What changes are included in this PR?
Implemented `handle_child_pushdown_result` for `SortExec` in
`datafusion/physical-plan/src/sorts/sort.rs`:
- For plain `SortExec` (no fetch) in the `Pre` phase: any filters not
absorbed by the child are collected into a new `FilterExec` inserted between
the `SortExec` and its child.
- For `SortExec` with fetch (TopK) or non-`Pre` phases: filters are not
absorbed, preserving correct TopK semantics where filtering after limiting
would change results.
## Are these changes tested?
Added comprehensive tests in
`datafusion/core/tests/physical_optimizer/filter_pushdown.rs` covering:
- Filter pushdown through sort into a scan that supports pushdown
- Filter pushdown through sort when scan does NOT support pushdown
(FilterExec inserted between sort and scan)
- Multiple conjunctive filters pushed through sort
- Filter NOT pushed through sort with fetch (TopK)
- Filter with projection pushed through sort
- Filter pushdown preserving the `preserve_partitioning` flag
- Filter with fetch limit propagated to SortExec (TopK conversion)
## Are there any user-facing changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
