neilconway opened a new pull request, #24127: URL: https://github.com/apache/datafusion/pull/24127
## Which issue does this PR close? - Related to #23982 ## Rationale for this change In Linear mode, BoundedWindowAggStream's evaluation sweep visits every live partition for every window expression on every input batch. A partition can be safely skipped if it received no new rows and already row currently in the partition has its output fully computed. This avoids a bunch of redundant work: re-evaluating the window function arguments and ORDER BY columns against the retained batch, building an empty result array, and other bookkeeping. This is particularly expensive for workloads with many partitions where only a few of those partitions receive rows in a given batch, as in the "32k sparse" benchmark below. Benchmarks: - linear / range / single / 100 dense: 42.3 ms -> 42.0 ms (~noise) - linear / range / single / 10000 dense: 158.7 ms -> 152.5 ms (-3.9%) - linear / range / single / 32768 sparse: 161.1 ms -> 108.0 ms (-33.0%) - linear / rows / single / 10000 dense: 132.0 ms -> 127.5 ms (-3.4%) - linear / range / multi / 10000 dense: 255.9 ms -> 236.4 ms (-7.6%) - sorted / range / single / 10000: 33.1 ms -> 33.7 ms (~noise) ## What changes are included in this PR? * Skip evaluating window expressions for fully calculated partitions * Add test case * Add assert checking that per-window-agg and per-partition state is consistent ## Are these changes tested? Yes. Existing tests pass. Added a new test to verify that "evaluate partition -> skip partition -> evaluate partition" sequence results in resuming accumulator states appropriately. I also checked that if the `is_end` conjunct is removed from the skip condition, the new assert added above fires and catches the bug. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
