Dandandan commented on PR #21328: URL: https://github.com/apache/datafusion/pull/21328#issuecomment-4186938218
> Do you expect any performance change due to this PR? Not really - `SortPreservingMergeExec` is almost always on the root of the plan, so it will be executed on the first poll (directly after creating it, instead of using `execute`). In cases it would do more than once in a plan (e.g. union of 2 `SortPreservingMergeExec`) it could actually slow things down as it will no longer run both in parallel. CoalescePartitions also does the eager spawning, and in queries with non-optimal concurrency, this leads to more parallelism But in my opinion we want to avoid this kind of implicit pipeline parallelism and use morsel-based parallelism do improve / reduce memory usage + (data/instruction) cache friendliness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
