sunchao opened a new pull request, #5376: URL: https://github.com/apache/datafusion-comet/pull/5376
## Why are the changes needed? Closes #5270. Spark compares flat floating-point `arrays_overlap` elements with boxed `Float`/`Double` equality: all NaN signs and payloads are equal, but `+0.0` and `-0.0` remain distinct. Comet currently compares the raw IEEE bit patterns instead, so a runtime-generated noncanonical NaN can incorrectly produce `false` or `NULL`. ## What changes were proposed in this PR? - Canonicalize NaN keys for both `Float32` and `Float64` in the existing flat-array fast path without changing signed-zero keys. - Preserve the existing nested-array comparison path; this change does not overlap with #5235. - Add Rust regressions covering positive, negative, and signaling NaN payloads; signed zeros; null semantics; and both nested-scan and hash-probe implementations. - Add a Parquet-backed Spark parity regression that generates noncanonical NaNs through runtime unary negation, checks both float widths, and verifies signed-zero, null, reversed-argument, and long-array behavior. ## How was this PR tested? - `cd native && cargo fmt --all -- --check` - `cd native && cargo test -p datafusion-comet-spark-expr --lib array_funcs::arrays_overlap::tests -- --nocapture` (23 tests passed) - `cd native && cargo build` - `JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home ./mvnw test -Pspark-3.5 -Dtest=none '-Dsuites=org.apache.comet.CometArrayExpressionSuite floating-point NaN payloads and signed zeros' -Dscalastyle.skip=true` - `JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home ./mvnw clean test -Pspark-4.0 -Dtest=none '-Dsuites=org.apache.comet.CometArrayExpressionSuite floating-point NaN payloads and signed zeros' -Dscalastyle.skip=true` - Maven Spotless checks passed for both Spark profiles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
