This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 6247f5b39a1c [SPARK-51789][CORE] Respect spark.api.mode and
spark.remote properly when parsing arguments in Spark Submission
6247f5b39a1c is described below
commit 6247f5b39a1c393d648a37dc810fb04a633c0f98
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Mon Apr 14 12:53:18 2025 +0900
[SPARK-51789][CORE] Respect spark.api.mode and spark.remote properly when
parsing arguments in Spark Submission
### What changes were proposed in this pull request?
This PR proposes to respect `spark.api.mode` and `spark.remote` properly
when parsing arguments in Spark Submission. Currently, the `isRemote` is set
early before starting to parse the configuration arguments at `getApiMode`.
### Why are the changes needed?
In Spark 4.0 release (Spark Connect distribution),
```bash
SPARK_CONNECT_MODE=1 ./bin/pyspark --conf spark.api.mode=classic --master
local
```
It fails as below:
```
/.../spark/python/pyspark/shell.py:94: UserWarning: Failed to initialize
Spark session.
warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
File "/.../spark/python/pyspark/shell.py", line 89, in <module>
spark = SparkSession._create_shell_session()
File "/.../spark/python/pyspark/sql/session.py", line 1249, in
_create_shell_session
return SparkSession._getActiveSessionOrCreate()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/.../spark/python/pyspark/sql/session.py", line 1260, in
_getActiveSessionOrCreate
spark = SparkSession.getActiveSession()
File "/.../spark/python/pyspark/sql/utils.py", line 357, in wrapped
return f(*args, **kwargs)
File "/.../spark/python/pyspark/sql/session.py", line 747, in
getActiveSession
if jSparkSessionClass.getActiveSession().isDefined():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: 'JavaPackage' object is not callable
```
We should fix this.
### Does this PR introduce _any_ user-facing change?
No to end users because the main change has not been released yet.
### How was this patch tested?
Manually tested with some combinations below:
Positive cases:
```
SPARK_CONNECT_MODE=1 ./bin/pyspark --conf spark.api.mode=classic --master
local
SPARK_CONNECT_MODE=1 ./bin/pyspark --master local --conf
spark.api.mode=connect
SPARK_CONNECT_MODE=1 ./bin/pyspark --conf spark.api.mode=classic
SPARK_CONNECT_MODE=1 ./bin/pyspark --conf spark.api.mode=connect
SPARK_CONNECT_MODE=1 ./bin/pyspark
```
Negative cases:
```
SPARK_CONNECT_MODE=1 ./bin/pyspark --conf spark.remote="local[*]" --conf
spark.api.mode=connect --conf spark.master="local[*]"
SPARK_CONNECT_MODE=1 ./bin/pyspark --master "local[*]" --remote "local[*]"
SPARK_CONNECT_MODE=1 ./bin/pyspark --conf spark.remote="local[*]" --conf
spark.api.mode=connect --master "local[*]"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50575 from HyukjinKwon/SPARK-51789.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit 0f5c98d3f3bfdb5d3c1736c10dfc58afa733e229)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
.../java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
index 26c14689fa7b..8eaefc6364a9 100644
---
a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
+++
b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
@@ -151,9 +151,6 @@ class SparkSubmitCommandBuilder extends
AbstractCommandBuilder {
OptionParser parser = new OptionParser(true);
parser.parse(submitArgs);
this.isSpecialCommand = parser.isSpecialCommand;
- if (conf.containsKey("spark.remote") ||
"connect".equalsIgnoreCase(getApiMode(conf))) {
- isRemote = true;
- }
} else {
this.isExample = isExample;
this.isSpecialCommand = true;
@@ -553,6 +550,10 @@ class SparkSubmitCommandBuilder extends
AbstractCommandBuilder {
String[] setConf = value.split("=", 2);
checkArgument(setConf.length == 2, "Invalid argument to %s: %s",
CONF, value);
conf.put(setConf[0], setConf[1]);
+ // If both spark.remote and spark.mater are set, the error will be
thrown later when
+ // the application is started.
+ isRemote |= conf.containsKey("spark.remote");
+ isRemote |= "connect".equalsIgnoreCase(getApiMode(conf));
}
case CLASS -> {
// The special classes require some special command line handling,
since they allow
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
