LiaCastaneda commented on issue #22758: URL: https://github.com/apache/datafusion/issues/22758#issuecomment-4937196287
@avantgardnerio are you refering to this PR? https://github.com/apache/datafusion/pull/22834 -- I took a look at it, the approach is interesting and something perhaps people can implement in their own code as a workaround to the DF memory accounting. Have you thought about integrating this with the DF `MemoryPool` somehow? I suppose we will need per query attribution for that. iiuc your PR, right now the guard tracks a single global balance and kill one query by panicing when it cannot alloc (which makes it possible to kill only that query and not all in the node), but it doesn't actually keep track of which query it killed nor which query was responsible for the pressure. To target the heavy query (instead of just whichever one happened to allocate when memory ran out) I guess you'd need per-query accounting. Maybe we would need a query id carried as a task_local var that propagates into every spawned task, and have the allocator read it per alloc or somthing simillar. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
