sunchao opened a new pull request, #5420: URL: https://github.com/apache/datafusion-comet/pull/5420
## Why are the changes needed? Closes #5418. Spark initializes AVG buffers with a zero sum and adds partial sums without coalescing nulls. Comet's empty scalar partial currently exports a null sum instead. In the supported native-partial / Spark-final path, empty scan partitions can therefore turn a nonempty AVG result into `NULL`. Decimal AVG must also distinguish an empty buffer from an overflow marker. Changing empty state to zero must not revive an overflowed accumulator or make an all-empty final aggregate divide by zero. ## What changes were proposed in this PR? - Export a typed zero sum and zero count for empty floating-point and decimal scalar AVG partials, including when no input batch arrives. - Preserve decimal overflow markers across state export, partial merges, and later update batches. Handle grouped overflow states with null counts, while keeping empty final results `NULL` and using errors in ANSI mode versus `NULL` in legacy/TRY mode for overflow. - Add native regressions for empty/all-null inputs, empty/nonempty merges, and overflow-state preservation, plus a multi-partition Spark-final regression for integer and narrow-decimal AVG. This does not change aggregate mixed-execution eligibility. The separate planner-safety issue is tracked in #5419. ## How was this PR tested? - Verified fail-before behavior on public `main`: the new Spark regression returned `[NULL, NULL]` instead of `[2.0, 11.000000]`; four of the eight new native tests also failed. - `cargo test -p datafusion-comet-spark-expr --lib`: **646 passed**. - Spark 4.0.4 / Java 17, `CometAggregateSuite` filtered with `-Dsuites='org.apache.comet.exec.CometAggregateSuite avg'`: **14 passed**, including empty input, mixed execution, decimal overflow, and TRY coverage. - Root-reactor Maven package, Spotless, Scala style checks, `cargo fmt --all -- --check`, and `git diff --check` passed. - Independent standalone-JAR replay on stock Spark 4.0.2: **20/20** synthetic cases match Spark with AQE off/on. The unmodified public-base JAR differs in eight of those cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
