2010YOUY01 commented on PR #23598: URL: https://github.com/apache/datafusion/pull/23598#issuecomment-4978197547
> Since #23032 remove `concat_batches` usages completely I am going to close this PR Thank you. I think the tradeoff between this PR and #23032 is: - This PR requires us to dig deeper into the memory allocator and possibly the OS to determine how to reduce RSS to 0.5×. - #23032 is guaranteed to eliminate the 2× memory amplification, but it adds complexity to many critical operators that are already quite complex. Further reducing memory usage with this PR’s approach may be challenging. I did a quick experiment before, and the measured RSS was not as low as 0.5×; if I remember correctly, my result was similar to yours. One possible reason is that the memory allocator caches released memory for future reuse. Therefore, to pursue this PR’s approach, we may need to better understand and tune the memory allocator—or possibly even OS-level behavior. If anyone knows how to do it, this PR's approach would be better (no extra complexity will be introduced to operators); otherwise I plan to help with #23032 later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
