andygrove opened a new pull request, #7421: URL: https://github.com/apache/datafusion-comet/pull/7421
## Which issue does this PR close? Part of #7406. ## Rationale for this change With `spark.comet.exec.aggregate.skipPartial.enabled`, a native partial aggregate can stop aggregating once its input looks mostly distinct and send the rest of the task's rows on as partial states. Comet only allows this when every aggregate function is a single-argument COUNT, so a query such as TPC-H q17, which takes an AVG grouped by a nearly unique key, keeps aggregating. In #7289 this change cut q17 at SF1000 from about 277 s to 242 s and stopped an executor from being OOM-killed. That PR runs the query on Comet Native Executors, but the gate is shared, so the change helps Comet on Spark too. This splits it out of #7289. ## What changes are included in this PR? - `AvgGroupsAccumulator`, the Float64 average that Comet uses for every non-decimal AVG (and for Spark's rewrite of a low-precision decimal AVG to `avg(UnscaledValue(x))`), implements `convert_to_state`. It used to return `NotImplemented`. Each row's state is its value and a count of one. A null or filtered-out row gets a zero sum and a zero count, which is the state `state()` emits for such a group. - `configure_skip_partial_aggregation` also admits a single-argument AVG whose result type is a double. A decimal AVG runs as `AvgDecimal`, whose `convert_to_state` is still not implemented, so it stays excluded. - The setting's description now lists AVG. ## How are these changes tested? - A new unit test in `avg.rs` checks that `convert_to_state` gives the same states as aggregating each row into its own group, with a null row and a filtered-out row. - `skip_partial_eligibility_is_fail_closed` now covers a double AVG (admitted), an AVG with no child, and a decimal AVG (both rejected). - `skip partial aggregation admits only supported native shuffle plans` in `CometAggregateSuite` checks that a double AVG over nullable columns skips and still matches Spark, and that a `DECIMAL(30, 2)` AVG does not skip. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
