andygrove opened a new pull request, #21967: URL: https://github.com/apache/datafusion/pull/21967
## Which issue does this PR close? - Partially addresses https://github.com/apache/datafusion/issues/14044 ## Rationale for this change Donate the `xxhash64` hash function from Comet so that other projects can benefit from it. The function was initially implemented in Comet by @advancedxy. This is a continuation of #19627, which has gone stale. To keep the change focused and easier to review, this PR adds only `xxhash64`. Murmur3 will follow in a separate PR. ## What changes are included in this PR? - Add `xxhash64(expr1, expr2, ...)` to `datafusion-spark`. - Add a shared `create_hashes_internal!` macro in `datafusion/spark/src/function/hash/utils.rs` for type dispatch (covering primitives, strings, binary, decimals, timestamps, dates, and the complex types `List`, `LargeList`, `FixedSizeList`, `Struct`, `Map`). Dictionary types are dispatched separately via turbofish in `xxhash64.rs`. - Add Rust unit tests for primitives, boundary values, emoji/CJK strings, float `-0.0` normalization, dictionaries (with and without nulls), `FixedSizeBinary`, `Struct`, and `List`. - Add sqllogictest coverage in `datafusion/sqllogictest/test_files/spark/hash/xxhash64.slt` with values verified against Spark. ## Are these changes tested? Yes, both Rust unit tests and sqllogictest are included. ## Are there any user-facing changes? A new `xxhash64` scalar function is available in `datafusion-spark`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
