avantgardnerio commented on issue #22758: URL: https://github.com/apache/datafusion/issues/22758#issuecomment-4937271645
@LiaCastaneda that's the 2nd PR in my series. In [this one](https://github.com/apache/datafusion/pull/22626), I demonstrated something closer to what you are suggesting (but it was only for testing). Unfortunately we can only kill whatever unlucky query goes over the limit (victim), but my intention was that when a kill was detected, we could use the memory tracker to sort queries by amount of tracked memory descending, and kill the perpatrator as well. > Maybe we would need a query id carried as a task_local var that propagates into every spawned task, and have the allocator read it I think that's what's in my OomGuard PR. It's definitely part of what we run at Coralogix. If the tokio task that gets killed is one we spawned, we know the query ID. Unfortunately operators like RepartitionExec spawn their own tasks with their own default panic handlers and those sadly just die. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
