ariel-miculas commented on PR #25372: URL: https://github.com/apache/datafusion/pull/25372#issuecomment-5696916106
> Currently the behaviour is to only keep the current batch reserved per cursor, meaning we don't actually reserve all the rows we are keeping owned. This leads to a lot of invisible memory, that is held, but not accounted by the pool. I think this should be explained in more detail. Specifically, from what I can tell, the existing issue is that we release the memory reservation when the cursor for a partition is dropped, ignoring the fact that ReusableRows keeps the memory alive. Other than that, the [cursors Vec](https://github.com/apache/datafusion/blob/0646a310cdeb25ba6a091e643dddd06da7ea181e/datafusion/physical-plan/src/sorts/merge.rs#L90) should keep the memory reservations alive for each input partition, and when the stream is polled, the previous reservation would be dropped and a new reservation created. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
