dwsmith1983 commented on PR #6810: URL: https://github.com/apache/datafusion-comet/pull/6810#issuecomment-6220916300
@andygrove one thing about the guard's comment, which says the fallback can go once apache/datafusion#26226 is fixed in a release Comet uses. The fix (apache/datafusion#26228, approved) makes the hash map path keep build order, but DataFusion still doesn't promise it: `HashJoinExec` reports the build side as unordered, and the maintainers would rather not commit to that order yet because a spilling hash join couldn't keep it (see [this comment](https://github.com/apache/datafusion/pull/26228#issuecomment-6207937548)). The guarantee is tracked in apache/datafusion#26235. So removing the fallback after the upgrade would leave Comet depending on behavior DataFusion doesn't promise, and a future spilling join could break it without any error. Should the guard wait for apache/datafusion#26235 instead? Since this PR is in the merge queue, I could point the comment at that issue in a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
