asolimando commented on PR #21330: URL: https://github.com/apache/datafusion/pull/21330#issuecomment-4182305248
It's a little slower but the spill to disk goes from 600MB to almost nothing, I guess there are scenarios that could trade some latency for no spilling. Unless the new Morsel approach improves on both latency and spilling, I wonder if we shouldn't make this option available via a configuration option (disabled by default)? WDYT @Dandandan? cc: @gene-bordegaray, FYI in case it might be relevant for the distributed DF scenario you are working on, which heavily uses RepartitionExec too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
