comphead commented on PR #6121:
URL:
https://github.com/apache/datafusion-comet/pull/6121#issuecomment-5783775150
An alternative to the diagram in this PR, covering the four points above.
```mermaid
%%{init: {'flowchart': {'wrappingWidth': 400}}}%%
flowchart LR
subgraph CG["Executor container, cgroup memory.max = executor.memory +
executor.memoryOverhead + memory.offHeap.size"]
direction TB
subgraph NAT["Native heap, allocated by Rust in this process"]
direction TB
NRES["Declared reservations<br>ExternalSorter, grouped aggregate, hash
and SMJ joins, ShuffleRepartitioner<br>seen by the pool and by
native_allocated"]
NUND["Undeclared Rust allocations<br>expression kernels, array
builders, Parquet page and decompression buffers,<br>per-plan FileMetadataCache
at 50 MiB each, object_store, tokio, FFI batches in flight<br>seen only by
native_allocated"]
NOPAQ["Outside Rust's GlobalAlloc<br>libzstd, aws-lc-sys, libhdfs,
jemalloc retained pages, fragmentation, Arrow padding<br>seen by nothing but
RSS"]
end
subgraph OFF["JVM off-heap, allocated by Unsafe and Java Arrow"]
direction TB
TUNG["Spark Tungsten pages"]
JSH["Comet shuffle pages<br>CometUnifiedShuffleMemoryAllocator"]
ARW["CometArrowAllocator<br>RootAllocator with no limit"]
end
subgraph HEAP["JVM heap, bounded by spark.executor.memory"]
direction TB
UNIF["Spark unified region<br>spark.memory.fraction, default 0.6"]
USER["User memory, the remaining 0.4<br>Comet plans, CometVectors,
iterators, Spark internals"]
end
subgraph REST["Neither heap"]
direction TB
NONHEAP["JVM non-heap<br>metaspace, code cache, JVM and tokio thread
stacks, Netty"]
PAGEC["Page cache from spill and shuffle files<br>charged to the
cgroup, reclaimable under pressure"]
end
end
GATE1["Comet gate, fair_unified only<br>rejects when pool total plus
request exceeds memory_limit / num_consumers<br>memory_limit is
memory.offHeap.size scaled by
spark.comet.exec.memoryPool.fraction<br>greedy_unified has no such gate"]
GATE2["Spark gate, one per task
attempt<br>TaskMemoryManager.acquireExecutionMemory<br>grants at most pool /
numActiveTasks, blocks below pool / 2 x numActiveTasks"]
OFFEX["Off-heap EXECUTION pool<br>one per executor, shared by every task
attempt"]
OFFST["Off-heap STORAGE pool<br>OFF_HEAP cached blocks,
spark.memory.storageFraction"]
ONPOOL["On-heap execution and storage pools"]
GCONLY["Bounded by the heap, budgeted by nobody"]
NOONE["No bound and no budget<br>spark.executor.memoryOverhead is the only
slack"]
NRES -->|"try_grow over JNI. One way only: NativeMemoryConsumer.spill
returns 0"| GATE1
GATE1 --> GATE2
TUNG --> GATE2
JSH --> GATE2
GATE2 --> OFFEX
OFFEX <-->|"borrows, evicting cached blocks"| OFFST
UNIF --> ONPOOL
USER --> GCONLY
NUND --> NOONE
NOPAQ --> NOONE
ARW --> NOONE
NONHEAP --> NOONE
PAGEC --> NOONE
classDef budgeted fill:#c7ecd0,stroke:#2f6b46,color:#000
classDef gate fill:#fff2b2,stroke:#8a7420,color:#000
classDef heaponly fill:#cfe4fb,stroke:#2f5680,color:#000
classDef unbudgeted fill:#f6c69a,stroke:#8a4b1f,color:#000
class NRES,TUNG,JSH,OFFEX,OFFST,UNIF,ONPOOL budgeted
class GATE1,GATE2 gate
class USER,GCONLY heaponly
class NUND,NOPAQ,ARW,NONHEAP,PAGEC,NOONE unbudgeted
```
What it changes. Two gate nodes put the fraction and the `1/numActiveTasks`
divisor where they actually apply. The off-heap pool becomes two boxes with a
borrow edge. The heap separates the unified region from user memory, which is
where Comet's own JVM objects live. The native heap is graded by which counter
can observe each part, so the picture doubles as a measurement guide for
`native_allocated` against `comet_memory_reserved_total`. The one-way
Comet-to-Spark relationship rides on an edge label rather than needing an
absent arrow.
If that is too much for one figure, the natural split is to move the gates
and the pool pair into a separate "what a single reservation passes through"
diagram under *Where Comet's budget comes from*, and leave this one
region-to-authority only.
I have not rendered the fence. `dev/ci/check-mermaid.py` is the authority
and it does not run on my machine either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
