suremarc commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5465612624
I think a more broadly useful concept is that of per-(logical-)partition orderings: i.e., the data is in time order when `<key>` is held constant. Then we should still be able to do streaming aggregation, while only needing to hold one time cell in memory per key. For the OP, you could go further and advertise the data is in `<key>, time` order when `date_bin(time)` is held constant, and that should allow for O(1) memory usage, similarly to the original proposal. I think this would be a natural extension of `EquivalenceProperties` and could also help in a number of other query shapes: * Time-partitioned, key-clustered data automatically becomes time order with a constant key filter (I added several ad-hoc fixes in DataFusion to enable this while working for Massive.com) * Window functions can operate on non-sorted data without a SortExec as long as it's in time order per partition * Streaming workloads can do streaming shuffles without a sort preserving repartition, keeping latency predictable. I no longer work for Massive.com, but I'd be curious if @xudong963 thinks this is a useful generalization for them, given that they also maintain a streaming engine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
