andygrove opened a new issue, #6028: URL: https://github.com/apache/datafusion-comet/issues/6028
The cgroup diagram in [memory_management](https://datafusion.apache.org/comet/contributor-guide/memory_management.html#what-the-container-sees), added in #6014, groups everything the container counts by who accounts for it. It does not say which configuration value sizes each region. That leaves the two facts the section goes on to state as prose only: 1. `spark.memory.offHeap.size` is part of the pod limit, not extra headroom on top of it. 2. `spark.executor.memoryOverhead` is the only slack in the container. Both are properties of how the configuration maps onto the regions, which is exactly the kind of thing a diagram carries better than a sentence. The mapping is also uneven in a way that is easy to get wrong: `spark.memory.offHeap.size` alone sizes three of the regions, including Comet's native reservations, which are neither off-heap in Spark's sense nor allocated by the JVM, while `spark.executor.memoryOverhead` sizes none of them. Adding the three configuration values as inputs to the existing diagram would make that visible without introducing a third diagram to the page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
