2010YOUY01 commented on PR #21425: URL: https://github.com/apache/datafusion/pull/21425#issuecomment-4275019173
I think the major considerations are 1. It is a blocking operation as stated in https://github.com/apache/datafusion/pull/21425#discussion_r3105561069 2. All spilling operators have to explicitly implement this method to clear the buffer Here is an alternative idea: allow the existing `reserve()` API to wait. Currently, for non-spillable consumers, a failed reservation immediately returns an error. Instead, we could let it wait for a period of time; once memory becomes available, the operation would be re-scheduled and retried. This approach reuses the existing API to support similar behavior. The key difference from the proposed `reclaim()` API is that `reclaim()` enables operators to immediately free buffers, which is more complex to implement. As a first step, we could extend the semantics of `reserve()` instead—though this would likely require introducing a separate memory pool. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
