shehab-ali opened a new pull request, #23843: URL: https://github.com/apache/datafusion/pull/23843
### Motivation - Add targeted hash-join microbenchmarks to exercise high-fanout and single-hot-bucket join cases with long string keys and multi-column keys. - Simplify and speed up candidate-pair equality filtering by eliminating per-row type dispatch and by using a pre-built comparator that respects `NullEquality` semantics. ### Description - Add benchmark definitions `benchmarks/sql_benchmarks/hj/benchmarks/q24.benchmark` and `q25.benchmark` that run count(*) queries exercising a single-hot-bucket long string-key join and a skewed multi-column string-key join respectively. - Register Q24 and Q25 in `benchmarks/src/hj.rs` by adding two `HashJoinQuery` entries with appropriate `sql`, `build_size`, `probe_size`, and `isolate_partitioned_join` metadata. - Refactor `equal_rows_arr` in `datafusion/physical-plan/src/joins/utils.rs` to use a `JoinKeyComparator` built once per batch pair, removing the previous `eq_dyn_null`/`FilterBuilder` approach and related per-array dispatch. - Remove unused imports and dead code associated with the old equality implementation and add two unit tests (`test_equal_rows_arr_filters_candidate_pairs` and `test_equal_rows_arr_respects_null_equality`) to validate candidate-pair filtering and null-equality behavior. ### Testing - Ran unit tests for the joins utilities in the physical-plan crate with `cargo test` and the new tests `test_equal_rows_arr_filters_candidate_pairs` and `test_equal_rows_arr_respects_null_equality` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
