sunchao opened a new pull request, #5469: URL: https://github.com/apache/datafusion-comet/pull/5469
## Why are the changes needed? Follow-up to #4870. Spark treats all NaN representations as equal and greater than non-NaNs, and treats `-0.0` and `+0.0` as peers. Arrow's raw total ordering distinguishes their bits, so native `RANK` / `DENSE_RANK` cutoffs can drop rows Spark retains. A native regression fixture returns 8 rows in Spark but only 4 without this fix. Fixing peer equality alone is insufficient: with secondary ordering keys, a sort that distinguishes NaN payloads or zero signs can separate rows that should be peers. ## What changes were proposed in this PR? - Reuse `NormalizeNaNAndZero` for scalar `FLOAT` / `DOUBLE` sort expressions shared by Sort, Window, and WindowGroupLimit. - Normalize native range-shuffle boundaries with the same helper as incoming keys. - Preserve original output values, including NaN payloads and zero signs; normalize only comparison keys. - Add native and Spark regressions for both float widths, NaN signs/payloads, signed zeros, compound keys, batch boundaries, null ordering, and range routing. Re-enable the existing signed-zero SQL regression. - Document that nested floating values and the existing strict-floating-point fallback policy are unchanged. ## How was this PR tested? - `make core`; `cargo fmt --all -- --check`; `cargo clippy --all-targets --workspace -- -D warnings`. - `cargo test --workspace --lib`: 908 passed, 4 existing ignored tests. - Spark 4.0: all 73 selected tests passed (full `CometWindowExecSuite`, all WindowGroupLimit SQL cases, and existing sort fallback tests). - Spark 3.5: all 21 selected tests passed (new regressions, all WindowGroupLimit SQL cases, and existing sort fallback tests). - Spark 4.1: all 21 selected tests passed (new regressions, all WindowGroupLimit SQL cases, and existing sort fallback tests), including Scala style and formatting checks. - Negative controls with the planner fix removed: both new native regressions and both Spark 4.0 FLOAT/DOUBLE regressions fail; the fixed runs pass. Spark tests explicitly require native Sort and WindowGroupLimit and check original output bits. ## Which issue does this PR close? Closes #5468. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
