alamb commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5551710271
I have spent some time studying these diagrams and the proposals. Thank you @jayzhan211 . My overall thought is that it describes a great long term design, but I (really) worry about the complexity -- I think the diagrams are describing the kind of complexity I would expect for the most mature Hybrid Hash Join implementations that have been built up over many years. Maybe with the advent of coding agents, we could accelerate our path there, but I feel like it might be overwhelming complex. The other challenge I see with this design is that there are still corner cases where the join may fail due to OOM If we take a step back, I think the most important feature for users is to have joins complete in cases where memory is limited and the inputs can not be entirely buffered. Once we have that, I predict we will be asked for faster performance. Today, as I understand it, joins with limited memory can be achieved by setting sort-merge-join, but this has the downside that it is slower for joins that would not have spilled ([see `prefer_hash_join` settings](https://datafusion.apache.org/user-guide/configs.html#prefer-hash-join-bool-default-true)). Thus, in my mind, a good goal for this project to add Spilling hash join would be to remove the need to set the `prefer_hash_join` flag apriori and have the Join implementation more dynamically adapt to handle memory pressure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
