EmilyMatt opened a new pull request, #25372: URL: https://github.com/apache/datafusion/pull/25372
Currently the behaviour is to only keep the current batch reserved per cursor, meaning we don't *actually* reserve all the rows we are keeping owned. This leads to a lot of invisible memory, that is held, but not accounted by the pool. (This scales fairly linearly with the number of partitions, in a regular round-robin case, for example.) While the mem overhead was greatly reduced by @ariel-miculas 's https://github.com/apache/datafusion/pull/23619 and https://github.com/apache/datafusion/pull/23802 It is still a very problematic area in terms of memory visibility. (Round-robin is an extreme case by nature) This PR does *not* reduce the mem usage or anything, it just makes the memory *visible* and keeps everything that needs to be reserved - reserved. I believe in order to ensure robustness, once this is merged another PR will need to be made to cap the merge reservation to the peak of the ExternalSorter reservation (due to some behaviours I've seen with fair-type pools, which treat spillable and non-spillable operators differently) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
