dwsmith1983 commented on PR #26228: URL: https://github.com/apache/datafusion/pull/26228#issuecomment-6250056890
@jayzhan211 thanks, both suggestions look right to me. **RightAnti for Comet.** Spark only plans a null-aware anti join for a single key with no other condition, so the missing filter support shouldn't come up. Building on the broadcast subquery also gives a much smaller table than the streamed partition Comet builds on today. With #23957 merged, Comet would get its order from the probe side, which `maintains_input_order` already reports, so it needs nothing new from DataFusion. @andygrove would moving Comet's `NOT IN` to the null-aware RightAnti work for you? If so, I think #26235 can be closed and the opt-in flag isn't needed, which should also settle @2010YOUY01's concern. **Concatenating first.** Agreed, the output is the same. It removes the offset bookkeeping, uses the same concat step as the ArrayMap path, and frees the input batches before the map is allocated, so peak build memory should drop. I'd hash the concatenated batch in slices so the hash buffer stays at batch size rather than growing to the whole build side. If Comet switches, I'd rework this PR around that change. I'd drop the cross-batch `LeftAnti` order test so nothing reads as a promise, and keep the duplicate-key test, since the `HashJoinExec` docs already describe that match order. I'll post build-side benchmarks against `main`, and if they show no gain I'll close this PR instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
