andygrove commented on issue #6050: URL: https://github.com/apache/datafusion-comet/issues/6050#issuecomment-5744808837
Follow-up: #6053 is closed and #6054 replaces it. Checking whether the adjustment reaches the executor container changed the answer. `ResourceProfileManager` materializes the default `ResourceProfile` in its constructor and snapshots `spark.executor.memoryOverhead` into it, and both the Kubernetes and YARN paths size executors from that profile rather than re-reading the conf. On Spark 3.4, 3.5 and 4.0 that happens before `SparkContext` builds the plugin container, so the existing on-heap adjustment has never reached the container on those versions. It only works on 4.1 and 4.2, where the two lines are ordered the other way round. That makes option 3 from the description unattractive and option 1 the only one that works on every version, so #6054 deletes the adjustment and warns at startup when `spark.executor.memoryOverhead` is unset instead. It does change behavior on 4.1/4.2, where containers shrink by `spark.comet.memoryOverhead` unless the application sets the overhead itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
