andygrove commented on issue #5212: URL: https://github.com/apache/datafusion-comet/issues/5212#issuecomment-5850470495
Status as of main at bc4be3996 (2026-09-26). Positions refer to the list in the description. | Position | Status | |---|---| | 1, fair pool checks pool-wide usage | Fixed by #6205, which keeps a per-consumer ledger. The branch-1.0 backport is draft #6228 | | 2 and 3, task-shared pool leak and poisoned mutex | Fixed by #5494 | | 4, `toString` | Fixed by #5398 | | 5, native memory invisible to Spark | Fixed by #5408 | | 6, 7, 9 and 10, `CometDiskBlockWriter` cross-task spilling and races | Fixed by #5493, which made the writer list per task | | 8, double `releasePlan` | Fixed by #5494 | | 11, `getUsed` in on-heap mode | Fixed by #5516, and #6066 later replaced the allocator | | 12, peak memory across spills | Fixed by #5463 | | 13, native shuffle spill error path | Fixed by #5461 | | 14, unified pool overflow | The asymmetry is gone since #6128. The overflow branch still keeps the grant, but reaching it needs a `usize` overflow | | 15, on-heap allocator singleton | Moot since #6066 | | 16, `memory_limit()` | Leaving this as is. A finite limit would only turn off DataFusion's `PartialReduce` hash stream, which Comet never plans | | 17, fair pool lock held across JNI | Open in #5613 | | 18, one JNI round trip per grow and shrink | Open in #5383 | | 19, tracing double count | The double count is fixed. The per-batch registry lock remains, but only with tracing on | | 20, thread registry leak | Fixed by #5464 | The pools now have unit tests against a fake Spark, and #5856 added JVM tests for `CometTaskMemoryManager`. Under "Not investigated", #6066 made the first two items moot. The third is real: every native plan in a task reports through the first plan's `CometTaskMemoryManager`, so the non-zero usage warning comes from the wrong plan. That is now #6255. A new sweep over the same code turned up more: - #6252: integer `SUM` state is left out of the aggregate's reservation. - #6253: native windows reserve nothing. - #6254: a spilled final aggregate fails during its replay. - #6256: the JVM shuffle crashes with checksums disabled. - #6257: every refused reservation is logged as a warning. - #6258: the hash-based JVM shuffle reports its output as spill. - #6259: two settings are validated only on executors. - #6260: the usage log understates untracked memory while a pool is overcommitted. The warnings in #2453 turned out to come from #5504. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
