2010YOUY01 commented on issue #23197: URL: https://github.com/apache/datafusion/issues/23197#issuecomment-4806401319
Here is some additional context: I think we don't have to rely on `BoundedWindowAggExec` in the future. The reason is that it does not assume the input is fully sorted by the window partition/order keys, so it performs internal sorting. This adds both implementation complexity and execution overhead. Here is my guess on why such design decision was made before: because the original window aggregation implementation was fairly naive: it buffered all input before continuing execution. As a result, a downstream implementation was upstreamed to support streaming and out-of-order input, where assuming fully sorted input was not possible for their use cases. Given that goal, the additional complexity made sense because it improved memory efficiency. However, DataFusion's primary use case is still batched analytics. Under that assumption, we can require the window operator to consume input that has already been repartitioned and sorted by the window partition/order keys. This makes the implementation both simpler and faster, and we can achieve the same memory efficiency given bounded window frames. So if we continue investing in vectorization and better parallelization, we could potentially evolve `WindowAggExec` directly. Its current implementation is relatively minimal, making it easier to extend, and eventually we could replace `BoundedWindowAggExec`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
