Dandandan commented on PR #2308: URL: https://github.com/apache/datafusion-ballista/pull/2308#issuecomment-5308605130
> In my experiments today, I was being able to recover most of the performance difference between ballista and datafusion-distributed mostly (after enabling AQE + some other configs, so it can use hash join / broadcast join which in itself gives 2-3x improvement) by keeping shuffle data in memory (RAM disk), yielding roughly -40% with a 4GiB RAM disk (e.g. keeping almost everything in memory). > > This suggests we should have an option to keep shuffle data in memory, and only spill once it exceeds a configured buffer, instead of always writing to disk. My bad, turned to be false info (queries were failing after first iteration after filling the disk). Although I think there is some ways to avoid / improve IO still. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
