dwsmith1983 opened a new pull request, #26228: URL: https://github.com/apache/datafusion/pull/26228
## Which issue does this PR close? - Closes #26226. ## Rationale for this change A `HashJoinExec` that builds a hash map put whole build batches in reverse order, so a `CollectLeft` anti or semi join that returns build rows emitted them last batch first. The array map path already kept input order, so the output order depended on how sparse the join keys were. Comet relies on the build side's order for a null-aware anti join and gets unsorted partitions when keys are sparse and the build side has more than one batch. ## What changes are included in this PR? As suggested in https://github.com/apache/datafusion-comet/pull/6810#discussion_r4248180112: - Batches are still inserted into the hash map last to first, so each key's chain keeps its earliest row on top. - Each batch now gets the offset of its start in forward order, and the build batches are concatenated forward, in both the ordinary and prepared build modes. - `concat_build_batches` loses its `reverse` flag, which no caller needs anymore. - The doc comment and diagram on `HashJoinExec` show the new layout. ## What is the testing strategy for this PR? New tests: - `join_left_anti_keeps_build_order_across_batches`: a `LeftAnti` join over two build batches (keys `[10, 20]` and `[3000, 4000]`) with no matches must return `10, 20, 3000, 4000`. It fails on main for every batch size on the hash map path (returns `3000, 4000, 10, 20`) and passes on the array map path. - `join_inner_keeps_build_order_for_duplicate_keys_across_batches`: duplicate keys across build batches still match in build order. This passes before and after; it pins the chain order so a future change can't flip it. - `prepared_build_keeps_build_order_across_batches`: prepared build mode only supports inner joins, so this checks the concatenated build batch directly (`10, 20, 3000, 4000`) and then runs a join to confirm the hash map indices still point at the right rows. It fails on main with the build batch in `3000, 4000, 10, 20` order. `cargo test -p datafusion-physical-plan --lib` and the sqllogictests pass; no existing expected output changed. ## Are there any user-facing changes? Build-side rows from a hash join (anti, semi, and unmatched rows of outer joins in `CollectLeft` mode) now come out in input order across build batches. No API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
