[
https://issues.apache.org/jira/browse/SPARK-56274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-56274:
-----------------------------------
Labels: pull-request-available (was: )
> Fix redundant SparkConf set() call in SparkClusterSubmissionWorker
> ------------------------------------------------------------------
>
> Key: SPARK-56274
> URL: https://issues.apache.org/jira/browse/SPARK-56274
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: kubernetes-operator-0.9.0
> Reporter: Jiwon Park
> Priority: Minor
> Labels: pull-request-available
>
> In {{SparkClusterSubmissionWorker.getResourceSpec()}}, each entry in
> {{sparkConf}} is written to {{effectiveSparkConf}} twice per iteration:
> {code:java}
> for (Map.Entry<String, String> entry : confFromSpec.entrySet()) {
> effectiveSparkConf.set(entry.getKey(), entry.getValue()); // first set
> String value = entry.getValue();
> if ("spark.kubernetes.container.image".equals(entry.getKey())) {
> value = value.replace("{{SPARK_VERSION}}", sparkVersion);
> }
> effectiveSparkConf.set(entry.getKey(), value); // second set
> }
> {code}
> For the {{spark.kubernetes.container.image}} key, the first {{set()}} stores
> the raw (unsubstituted) value before the {{{{SPARK_VERSION}}}} placeholder is
> replaced, and the second {{set()}} immediately overwrites it with the correct
> value. For all other keys, both calls write the identical value. The first
> {{set()}} is therefore always redundant.
> *Fix:* Remove the first {{set()}} call and restructure the loop to call
> {{set()}} exactly once per entry with the correctly substituted value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
