andygrove commented on code in PR #6054:
URL: https://github.com/apache/datafusion-comet/pull/6054#discussion_r4057145491
##########
spark/src/main/scala/org/apache/spark/Plugins.scala:
##########
@@ -176,6 +149,38 @@ object CometDriverPlugin extends Logging {
}
}
+ // Comet's native allocations are made by the Rust global allocator and live
in the native heap.
+ // The share that operators reserve is charged against a memory pool, but
everything else --
+ // expression kernels and Arrow array builders, decompression buffers,
Parquet reader structures,
+ // object store buffers, the tokio runtime, allocator overhead -- is covered
by no budget at all,
+ // and neither is Comet's JVM-side Arrow allocator. The only slack the
executor container has for
+ // that is spark.executor.memoryOverhead, which the JVM's own non-heap usage
already draws on.
+ //
+ // Comet used to add spark.comet.memoryOverhead to it here, but a driver
plugin cannot: on Spark
+ // 3.4, 3.5 and 4.0, SparkContext builds the default ResourceProfile before
it creates the plugin
+ // container, and the cluster managers size executors from that profile
rather than re-reading
+ // the conf, so the new value never reached the container. Say so while the
application is still
+ // starting up instead, because this has to be set before the SparkContext
is created.
+ private[apache] def warnIfExecutorMemoryOverheadUnset(conf: SparkConf): Unit
= {
+ val cometEnabled = getBooleanConf(conf, CometConf.COMET_ENABLED)
+ val cometExecEnabled = getBooleanConf(conf, CometConf.COMET_EXEC_ENABLED)
+ val cometShuffleEnabled = getBooleanConf(conf,
CometConf.COMET_SHUFFLE_ENABLED)
+ val cometActive = cometEnabled && (cometExecEnabled || cometShuffleEnabled)
+
+ if (cometActive && !conf.contains(EXECUTOR_MEMORY_OVERHEAD.key)) {
Review Comment:
on-heap mode is only enabled for running Spark SQL tests locally and in CI.
On-heap mode should never be used for a YARN deployment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
