This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c3934193e262 [SPARK-53125][TEST] RemoteSparkSession prints whole
`spark-submit` command
c3934193e262 is described below
commit c3934193e2624b8a1f8c2541e27a932dbfd8360b
Author: Cheng Pan <[email protected]>
AuthorDate: Wed Oct 8 08:29:41 2025 -0700
[SPARK-53125][TEST] RemoteSparkSession prints whole `spark-submit` command
### What changes were proposed in this pull request?
Make RemoteSparkSession print the whole `spark-submit` command in debug
mode, which helps the developers to understand the SPARK_HOME, classpath, log
output, etc. of the testing connect server process.
### Why are the changes needed?
Improve the debug message.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
export SPARK_DEBUG_SC_JVM_CLIENT=true
```
```
sbt:spark-connect-client-jvm> testOnly *ClientE2ETestSuite -- -z "throw
SparkException with large cause exception"
[info] ClientE2ETestSuite:
Starting the Spark Connect Server...
Using jar:
/Users/chengpan/Projects/apache-spark/sql/connect/server/target/scala-2.13/spark-connect-assembly-4.1.0-SNAPSHOT.jar
Using jar:
/Users/chengpan/Projects/apache-spark/sql/catalyst/target/scala-2.13/spark-catalyst_2.13-4.1.0-SNAPSHOT-tests.jar
/Users/chengpan/Projects/apache-spark/bin/spark-submit \
--driver-class-path
/Users/chengpan/Projects/apache-spark/sql/connect/server/target/scala-2.13/spark-connect-assembly-4.1.0-SNAPSHOT.jar
\
--class org.apache.spark.sql.connect.SimpleSparkConnectService \
--jars
/Users/chengpan/Projects/apache-spark/sql/catalyst/target/scala-2.13/spark-catalyst_2.13-4.1.0-SNAPSHOT-tests.jar
\
--conf spark.connect.grpc.binding.port=15707 \
--conf
spark.sql.catalog.testcat=org.apache.spark.sql.connector.catalog.InMemoryTableCatalog
\
--conf spark.sql.catalogImplementation=hive \
--conf spark.connect.execute.reattachable.senderMaxStreamDuration=1s \
--conf spark.connect.execute.reattachable.senderMaxStreamSize=123 \
--conf spark.connect.grpc.arrow.maxBatchSize=10485760 \
--conf spark.ui.enabled=false \
--conf
spark.driver.extraJavaOptions=-Dlog4j.configurationFile=/Users/chengpan/Projects/apache-spark/sql/connect/client/jvm/src/test/resources/log4j2.properties
/Users/chengpan/Projects/apache-spark/sql/connect/server/target/scala-2.13/spark-connect-assembly-4.1.0-SNAPSHOT.jar
...
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #51846 from pan3793/SPARK-53125.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../apache/spark/sql/connect/test/RemoteSparkSession.scala | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/test/RemoteSparkSession.scala
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/test/RemoteSparkSession.scala
index 059b8827b4b6..efb6c721876c 100644
---
a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/test/RemoteSparkSession.scala
+++
b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/test/RemoteSparkSession.scala
@@ -71,7 +71,7 @@ object SparkConnectServerUtils {
findJar("sql/catalyst", "spark-catalyst", "spark-catalyst", test =
true).getCanonicalPath
val command = Seq.newBuilder[String]
- command += "bin/spark-submit"
+ command += s"$sparkHome/bin/spark-submit"
command += "--driver-class-path" += connectJar
command += "--class" +=
"org.apache.spark.sql.connect.SimpleSparkConnectService"
command += "--jars" += catalystTestJar
@@ -79,7 +79,14 @@ object SparkConnectServerUtils {
command ++= testConfigs
command ++= debugConfigs
command += connectJar
- val builder = new ProcessBuilder(command.result(): _*)
+ val cmds = command.result()
+ debug {
+ cmds.reduce[String] {
+ case (acc, cmd) if cmd startsWith "-" => acc + " \\\n " + cmd
+ case (acc, cmd) => acc + " " + cmd
+ }
+ }
+ val builder = new ProcessBuilder(cmds: _*)
builder.directory(new File(sparkHome))
val environment = builder.environment()
environment.remove("SPARK_DIST_CLASSPATH")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
