andygrove opened a new pull request, #6237: URL: https://github.com/apache/datafusion-comet/pull/6237
## Which issue does this PR close? No issue; this is documentation only. It supersedes #6121, which I'm closing. ## Rationale for this change The diagrams in #6121, and the one it was redrawing, try to show every memory consumer in the executor and who accounts for it. That makes them hard to follow, and they don't help an end user who just wants to know why Comet needs both `spark.memory.offHeap.size` and `spark.executor.memoryOverhead`. This starts over with a deliberately simple diagram that explains the concept: Comet shares the off-heap memory pool with Spark for its memory-hungry operators, and it also needs native memory outside the JVM that the pool doesn't track, so `spark.executor.memoryOverhead` has to have room for it. It is not meant to be complete or exact. For example, it draws Comet's reservations inside the off-heap pool, where they are charged, rather than in the native heap, where they live. The prose around it, and the detailed breakdown in the contributor guide, carry the precise version. ## What changes are included in this PR? - `docs/source/_static/images/comet-executor-memory.svg`: a new hand-written SVG ([rendered](https://github.com/andygrove/datafusion-comet/blob/972528c2731ea9887174837a33cee85c9c5cc2a1/docs/source/_static/images/comet-executor-memory.svg)). It shows the executor container as the JVM heap, the off-heap memory pool and the memory overhead, the config that sizes each, which of them Spark and Comet use, and two numbered callouts for the shared pool and the overhead. The colours follow the existing site diagrams, yellow for Spark and green for Comet. It is an SVG rather than mermaid for control over the layout, and it draws its own white background so it reads on both the light and dark site themes. - Tuning guide: the diagram goes at the top of Memory Tuning, after the paragraph listing the two things Comet needs configured, which the callouts match. - Contributor guide: the same diagram goes at the end of the Overview in `memory_management.md`, with a sentence tying it to the rest of the page. The detailed diagram under "What the container sees" is unchanged. ## How are these changes tested? Documentation only, no code paths touched. `npx prettier@latest --check` passes on both pages. The SVG is well-formed XML and was rendered in headless Chrome, including with a deliberately wide fallback font: an SVG embedded as an image cannot use the site's web fonts, so most readers get a system font. I have not built the site with Sphinx locally; the docs only build on push to main, and both image links use the same relative `_static/images` paths as `profiling.md` and `tracing.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
