neilconway opened a new pull request, #24148: URL: https://github.com/apache/datafusion/pull/24148
## Which issue does this PR close? - Related to #23982 ## Rationale for this change After ingesting a batch of data, updating accumulator state, and emitting new output rows, `BoundedWindowAggStream` prunes each partition to reclaim state that is no longer needed: `prune_out_columns` trims emitted results that are no longer needed, and `prune_partition_batches` drops buffered input rows that aren't needed by any window expression. Both functions did work proportional to the # of live partitions, despite pruning being a no-op for partitions that didn't receive rows in the most recent batch: - `prune_out_columns` looked up every partition's buffer by hashing its partition key and re-sliced every result column, including zero-length prunes that rebuilt an identical column. - `prune_partition_batches` put an entry in its prune-count map for every live partition, cloning each partition's key (a Vec<ScalarValue>); for sparse workloads (# of partitions > batch-size), most prune counts will be zero and this did a lot of redundant work. Restructure both passes to pass over quiet partitions: - `prune_out_columns` iterates the partition buffers and only processes partitions with a nonzero emitted-row count. Hash lookups now happen only for partitions that emitted rows since the previous pass. - `prune_partition_batches` only keeps partitions with positive prune counts in its map Benchmarks (after applying #24127): - linear / range / single / 100 dense: 43.2 ms -> 43.0 ms (~noise) - linear / range / single / 10000 dense: 156.4 ms -> 157.0 ms (~noise) - linear / range / single / 32768 sparse: 111.2 ms -> 86.3 ms (-22.4%) - linear / rows / single / 10000 dense: 133.2 ms -> 133.3 ms (~noise) - linear / range / multi / 10000 dense: 246.3 ms -> 245.9 ms (~noise) - sorted / range / single / 10000: 34.5 ms -> 34.3 ms (~noise) ## What changes are included in this PR? * Optimize window state pruning as described above * Update and clarify comments in several places ## Are these changes tested? Yes, covered by existing tests. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
