zhuqi-lucas commented on code in PR #21182:
URL: https://github.com/apache/datafusion/pull/21182#discussion_r3035082232
##########
datafusion/physical-optimizer/src/pushdown_sort.rs:
##########
@@ -95,7 +100,14 @@ impl PhysicalOptimizerRule for PushdownSort {
// Each node type defines its own pushdown behavior via
try_pushdown_sort()
match sort_input.try_pushdown_sort(required_ordering)? {
SortOrderPushdownResult::Exact { inner } => {
- // Data source guarantees perfect ordering - remove the
Sort operator
+ // Data source guarantees perfect ordering - remove the
Sort operator.
+ // Preserve the fetch (LIMIT) from the original SortExec
so the
+ // data source can stop reading early.
+ let inner = if let Some(fetch) = sort_exec.fetch() {
+ inner.with_fetch(Some(fetch)).unwrap_or(inner)
Review Comment:
Thanks @alamb, this is part of the new functionality in this PR, not a
standalone bug fix. Before this PR, PushdownSort still no feature which
returned Exact.
This PR adds a new Exact path via statistics-based file reordering — when
SortExec is eliminated, its fetch (LIMIT) must be preserved on DataSourceExec
so it can stop reading early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
