andygrove opened a new pull request, #5996:
URL: https://github.com/apache/datafusion-comet/pull/5996
## Which issue does this PR close?
Relates to #4576.
## Rationale for this change
The memory management guide describes the budgets, the allocators and the
accounting gap in prose
and tables, but there is no picture of how the regions actually sit inside
one container. The
relationship that causes the most confusion, that
`spark.memory.offHeap.size` is read twice by two
allocators that never compare notes, is stated in two separate sections and
easy to miss.
The page also did not record what a cgroup counter actually measures.
`memory.current` includes
reclaimable page cache, so on any workload that reads files it saturates at
the container limit
while the non-reclaimable part stays far below it. That matters because it
rules out the obvious
design for a container memory guard: thresholding on `memory.current` fires
on healthy queries, and
raising the threshold only delays the false trip. This was measured rather
than reasoned about, and
the result belongs next to the rest of the accounting discussion.
## What changes are included in this PR?
- A new `Memory layout` section with a diagram showing two views of the same
executor container:
what Spark configures, which is what sizes the pod request, and what the
kernel counts, which is
what the OOM killer acts on. Both total the same limit and the regions do
not line up.
- A third entry in the list of facts that are easy to get wrong under `What
the container sees`,
covering the page cache behaviour of `memory.current` and naming the
quantity that does predict a
kill.
- A matching entry under `Open problems`, so the lack of a usable
container-level pressure signal
sits alongside the other known gaps.
- Two occurrences of the word "haircut" replaced with "margin".
The figures come from a measured executor: TPC-H SF100 Q9, one executor at a
9 GiB pod limit,
cgroup v2. `memory.current` reached 9215 MB of 9216 MB within about twelve
seconds of startup and
stayed there for the whole query, while `anon` peaked at 5044 MB, 55 percent
of the limit.
## How are these changes tested?
Documentation only, no code changes.
`prettier --check` passes on the modified file. The three cross-reference
anchors were verified
against the headings they target, and the diagram's box rows were checked to
be of uniform width so
they render aligned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
