andygrove opened a new pull request, #6033:
URL: https://github.com/apache/datafusion-comet/pull/6033
## Which issue does this PR close?
Closes #6032.
## Rationale for this change
#6014 and #6029 established, in the contributor guide, where Comet's memory
actually lives and which configuration value sizes each part of the executor
container. Two of those facts are the ones users get wrong, and neither
appeared anywhere in the tuning guide: `spark.memory.offHeap.size` is part of
the container budget rather than headroom on top of it, and
`spark.executor.memoryOverhead` is the container's only slack while sizing no
budget any consumer can draw on.
The tuning guide opened the memory section with "specify how much memory
Comet can use in addition to memory already allocated to Spark", which reads as
though Comet's budget sits outside what the container already accounts for.
That is exactly the misconception the contributor-guide diagram was changed to
prevent, and the audience that needs it most does not read the contributor
guide.
The practical consequence was that the guide gave no way to choose between
the three settings that respond to an executor OOM kill, and they do not behave
alike. Raising `spark.executor.memoryOverhead` widens the container and leaves
Comet's budget alone. Lowering `spark.comet.exec.memoryPool.fraction` leaves
the container alone and shrinks what Comet may reserve. Raising
`spark.memory.offHeap.size` does both, which makes it the intuitive response
and the least reliable one, since it also raises the ceiling on the unaccounted
overshoot that caused the kill. The guide also never said what an OOM looks
like from each of the three budgets, so there was nothing to tell a user
whether the setting they were about to change was the one that overflowed.
## What changes are included in this PR?
All in `docs/source/user-guide/latest/tuning.md`:
- The Memory Tuning intro is rewritten to say Comet re-divides the
executor's memory rather than adding to it.
- A new **How Comet's Memory Fits in the Executor Container** section:
Spark's container sizing sum, the two facts stated against it, and a table
comparing how the three settings move the container and Comet's budget.
- A new **Diagnosing Out-of-Memory Failures** section: a table
distinguishing a container kill (exit 137 / `OOMKilled`), JVM heap exhaustion
(exit 52), and a task-level `SparkOutOfMemoryError`, with what to change in
each case and which settings show what the pool was doing.
- A note that `fair_unified` is currently stricter than the paragraph
describing it, pointing at #5961. The existing description is left in place
rather than rewritten, because #5961 is a regression: when it is fixed the note
is simply deleted.
- A link from the user guide to the contributor guide's Memory Management
page, which so far only linked one way.
The container sizing sum is Spark's own, from
`ResourceProfile.getResourcesForClusterManager`, and matches the one already
stated in the contributor guide.
One thing this PR deliberately does not touch: the TPC-H figures in
**Determining How Much Memory to Allocate** ("Comet requires at least 5 GB of
RAM") do not say which setting the number refers to, and I did not want to
guess at a reading of numbers I cannot re-derive. Worth a separate look.
## How are these changes tested?
Docs only, no code paths touched. No mermaid fences are added, so the #6021
pre-render step is unaffected.
`npx prettier "docs/source/user-guide/latest/tuning.md" --check` passes,
which is what the Preflight job runs.
The two new in-page anchors follow the slug form MyST generates with
`myst_heading_anchors = 4`, matching the existing
`#where-comets-budget-comes-from` link in `memory_management.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
