comphead commented on code in PR #6279: URL: https://github.com/apache/datafusion-comet/pull/6279#discussion_r4146848657
########## docs/source/user-guide/latest/tuning/operators.md: ########## @@ -59,6 +59,17 @@ boundaries. A shuffled hash join can still filter probe batches after shuffle, b its filter back to an earlier scan stage. Compare the [runtime-filter and scan metrics](../metrics.md#hash-joins) with the setting disabled to distinguish reduced hash-probe work from reader I/O savings. +## Window Functions + +`PERCENT_RANK`, `CUME_DIST`, `NTILE`, and aggregates whose frame ends at `UNBOUNDED FOLLOWING`, which includes an +aggregate over `PARTITION BY` with no `ORDER BY` such as `sum(x) OVER (PARTITION BY k)`, need a whole window partition +before they return anything. Comet buffers one window partition at a time for them and reserves it from its native +memory pool, but it does not yet provide spill-to-disk for them. A window partition that does not fit in its share of +the pool, for example because of a heavily skewed key or a window without `PARTITION BY`, fails the task with a memory Review Comment: This section describes a change in outcome. A window partition that does not fit its share now fails the task, where before it ran on untracked memory. Would it make sense to add an entry for it to the upgrade guide? `docs/source/contributor-guide/config_conventions.md` (Changing the Behavior of an Existing Config) asks for one, plus a `spark.comet.legacy.*` key. I'm not sure a key is expected for a memory accounting fix, so a note like the 1.1.0 "Memory Pool Limits" entry may be enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
