andygrove commented on code in PR #6051: URL: https://github.com/apache/datafusion-comet/pull/6051#discussion_r4054291395
########## docs/source/user-guide/latest/tuning.md: ########## @@ -132,6 +138,31 @@ need to spill or have a single spillable operator. [shuffle]: #shuffle [Advanced Memory Tuning]: #advanced-memory-tuning +### Configuring Executor Memory Overhead + +Enabling off-heap memory is not sufficient on its own. It is recommended to also set +`spark.executor.memoryOverhead` when running Comet: + +``` +spark.executor.memoryOverhead=2g Review Comment: Good catch. `calculateOverHeadMemory` is a `getOrElse`, not a max, so a flat `2g` is a reduction on anything past roughly a 20 GiB executor, and worse for a PySpark app on Kubernetes where the propagated factor is 0.4. I've reworked the section in d334434. It now describes the derivation accurately — the 0.1 factor, the 0.4 exception for PySpark and SparkR submitted to Kubernetes in cluster mode, and the fact that the 384 MiB floor became `spark.executor.minMemoryOverhead` in 4.0 — and says outright that setting the value replaces the derived one rather than adding to it. `2g` is now scoped to small executors, with your 4 GiB (409 MiB) and 32 GiB (3276 MiB) numbers as the contrast, and anything larger is pointed at `spark.executor.memoryOverheadFactor` so the overhead keeps scaling with the executor. I left the `2g` in the `spark-shell` examples alone since none of them set `spark.executor.memory`, so they're on the 1 GiB default and the derived overhead there is the 384 MiB floor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
