andygrove opened a new issue, #6260: URL: https://github.com/apache/datafusion-comet/issues/6260
### Describe the bug The executor's memory usage log prints `allocated` and `reserved`, and the tuning guide tells users to read the difference as native memory that no pool tracks. The container warning computes the same difference and adds Spark's off-heap usage to it ([CometExecIterator.scala#L557-L575](https://github.com/apache/datafusion-comet/blob/bc4be39964cbe9cdb5f2a949740a8164e6b5755b/spark/src/main/scala/org/apache/comet/CometExecIterator.scala#L557-L575)). `reserved` is the sum of the pools' `reserved()`. Since #6128 that total also includes bytes a `grow` recorded without a grant from Spark, the overcommit ([spark_memory.rs#L55-L67](https://github.com/apache/datafusion-comet/blob/bc4be39964cbe9cdb5f2a949740a8164e6b5755b/native/core/src/execution/memory_pools/spark_memory.rs#L55-L67)). Those bytes are real allocations, but they get subtracted as if Spark were accounting for them, and they aren't in Spark's off-heap usage either. So while any pool is overcommitted, the untracked figure and t he container warning both come out lower by the amount of the overcommit. ### Expected behavior The log subtracts only what Spark has actually granted, or reports the overcommit on its own line, so memory recorded without a grant still counts toward the container footprint. ### Additional context The overcommit is usually small and short-lived, so this is minor. #6250 changes the same formula, so the two should be reconciled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
