comphead opened a new pull request, #7139: URL: https://github.com/apache/datafusion-comet/pull/7139
## Which issue does this PR close? No issue filed. Related: apache/datafusion#26532 was found with these benchmarks, and apache/datafusion#26533 fixes it. ## Rationale for this change There is no benchmark for `CometWindowExec`. The native window operator also reports no time of its own in the Spark UI, so a slow window is hard to tell apart from the sort or shuffle around it. ## What changes are included in this PR? - `native/operators/benches/window.rs`: criterion benches for DataFusion's `BoundedWindowAggExec` built the way the planner builds it, over in-memory batches already sorted on the partition and order keys. Function names resolve to the window UDF when one exists and to the aggregate UDF otherwise, as in `find_df_window_function`. The groups vary the rows per PARTITION BY group, the function (`row_number`, `max`, `last`, `first(v, true)`, `last(v, true)`), the argument type (bigint or struct), and the number of carried columns. - `CometWindowBenchmark`: the same dimensions end to end, with three arms per case: Spark, Comet, and Comet with `spark.comet.exec.window.enabled=false`. Before a case is timed, every arm must return the same row and the Comet arm must run `CometWindowExec`. Results on an Apple M5 Max, 512K rows, `local[1]`, whole query: | Case | Spark | Comet | Comet, Spark window | |---|---:|---:|---:| | `last(struct, true)`, 1 row per partition, 18 carried columns | 1283 ms | 2119 ms | 1187 ms | | `last(struct, true)`, 4 rows per partition | 452 ms | 713 ms | 330 ms | | `last(v, true)`, 16384 rows per partition | 236 ms | 834 ms | 154 ms | | `max(v)`, 16384 rows per partition | 239 ms | 163 ms | 155 ms | The last case at 16384 rows per partition is the quadratic scan fixed in apache/datafusion#26533. The small-partition cases point at per-row and per-partition costs in the DataFusion operator. ## How are these changes tested? These are benchmarks, not tests. Both compile and pass `cargo clippy --all-targets -- -D warnings`, rustfmt and spotless, and both were run locally to produce the results above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
