2010YOUY01 commented on PR #26228:
URL: https://github.com/apache/datafusion/pull/26228#issuecomment-6207937548
I think this is both an optimization opportunity and a potential constraint
on future features.
For example, if HJ runs out of memory, the ordering guarantee could prevent
it from switching to a spilling implementation that doesn't preserve that order:
```text
-- Example: Top-K query ordered by a column from the HJ build side
-- Optimizer exploits the join's ordering guarantee to eliminate SortExec
LimitExec(order_by = build_table, k=3)
HashJoinExec
-- If HJ needs to spill, the ordering guarantee no longer hold.
-- The fallback would require changing the surrounding plan shape (not
easily doable now)
LimitExec
SortExec
HashJoinExec
```
In general, I think we should prioritize general-purpose features over
specialized optimizations. Perhaps we should first figure out how to support
this kind of fallback cleanly before committing to the ordering guarantee.
(though I'm also unsure if ordering promise itself is a common requirement that
worth maintaining)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
