andygrove commented on PR #6835: URL: https://github.com/apache/datafusion-comet/pull/6835#issuecomment-6126048778
@dwsmith1983, in my last comment I said the plan stays the same, but I'd like to revisit it. This branch had to wait for #6810 because it turned the wrong order above a NOT IN into wrong groups. 2f95eefb36 removed that: `keeps_jvm_input_order` stops at any hash join that isn't an inner join, so an outer join above the null-aware anti join no longer declares an order over it, and your NOT IN test passes here. Once #6810 sorts above the NOT IN, `with_declared_order` sees the sort's order and leaves it alone, so either landing order ends in the same place. What still ties the two PRs together is field 11 and the lines both of them change in `CometHashJoin.doConvert` and the planner's hash join. Would you be OK with this one landing first? It only needs review and the Spark SQL run now, and since you're reworking #6810 for the NOT IN case anyway, you could build that on top of this branch rather than me rebasing onto yours afterwards. If it does go first, the narrowed #6810 could reuse `streamed_sort_orders` instead of adding a field. The null-aware anti join is always a left anti join built on the right, so its output is the streamed side's columns, and the same expressions work for a sort above the join. The difference is that the sort needs the whole order, where this branch sends the longest prefix it can express, so the NOT IN case would need to fall back when any key can't be expressed. Field 12 works too if you'd rather keep them separate. If you'd prefer the original order, that's fine, and I'll rebase as planned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
