2010YOUY01 commented on issue #21381: URL: https://github.com/apache/datafusion/issues/21381#issuecomment-4196015885
> Thanks [@2010YOUY01](https://github.com/2010YOUY01) for the context! This issue is specifically about the case after SortExec is eliminated via sort pushdown ([#21182](https://github.com/apache/datafusion/pull/21182)). When sort elimination removes SortExec entirely, there is no Stage 1 merge. SPM becomes the only merge operator, reading directly from I/O-bound DataSourceExec partitions. > > I've updated the issue description to clarify in case any misunderstanding. > > In that scenario, a parallel merge within SPM itself (splitting N input streams into groups and merging in parallel) could help. The DuckDB parallel k-way merge link is great — thanks for sharing! Ah, I see — that's a good optimization opportunity. Would you mind if I work on this, if you haven’t started on it yet? I’ve been thinking about improving the merge strategy similarly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
