andygrove commented on code in PR #6279: URL: https://github.com/apache/datafusion-comet/pull/6279#discussion_r4303087935
########## docs/source/user-guide/latest/tuning/operators.md: ########## @@ -59,6 +59,17 @@ boundaries. A shuffled hash join can still filter probe batches after shuffle, b its filter back to an earlier scan stage. Compare the [runtime-filter and scan metrics](../metrics.md#hash-joins) with the setting disabled to distinguish reduced hash-probe work from reader I/O savings. +## Window Functions + +`PERCENT_RANK`, `CUME_DIST`, `NTILE`, and aggregates whose frame ends at `UNBOUNDED FOLLOWING`, which includes an +aggregate over `PARTITION BY` with no `ORDER BY` such as `sum(x) OVER (PARTITION BY k)`, need a whole window partition +before they return anything. Comet buffers one window partition at a time for them and reserves it from its native +memory pool, but it does not yet provide spill-to-disk for them. A window partition that does not fit in its share of +the pool, for example because of a heavily skewed key or a window without `PARTITION BY`, fails the task with a memory Review Comment: Good call. I added a "Window Functions" entry to the 1.2.0 section of the upgrade guide, in the style of the 1.1.0 "Memory Pool Limits" entry. I didn't add a `spark.comet.legacy.*` key. The old behavior was memory that nobody tracked, so the only way back is to turn native windows off, and the entry says that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
