This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 146c2baebd84 [SPARK-54085][CORE] Fix `initialize` to add `CREATE`
option additionally in `DriverRunner`
146c2baebd84 is described below
commit 146c2baebd84dee54c24233ecb9013493c76b38b
Author: EricGao888 <[email protected]>
AuthorDate: Thu Oct 30 08:45:58 2025 -0700
[SPARK-54085][CORE] Fix `initialize` to add `CREATE` option additionally in
`DriverRunner`
### What changes were proposed in this pull request?
Fix the usage of Files.writeString in DriverRunner of Standalone Worker.
Currently users will get errors below when submitting jobs to standalone
cluster with restful api:
```
25/10/30 16:02:59 INFO DriverRunner: Killing driver process!
25/10/30 16:02:59 INFO CommandUtils: Redirection to
/Users/ericgao/workspace/open_source/spark/work/driver-20251030160259-0020/stdout
closed: Stream closed
25/10/30 16:02:59 WARN Worker: Driver driver-20251030160259-0020 failed
with unrecoverable exception: java.nio.file.NoSuchFileException:
/Users/ericgao/workspace/open_source/spark/work/driver-20251030160259-0020/stderr
```
### Why are the changes needed?
With only StandardOpenOption.APPEND:
If the target file already exists, the string is appended to the end of the
file.
If the target file does not exist, the call throws NoSuchFileException
because append requires an existing file.
With StandardOpenOption.CREATE, StandardOpenOption.APPEND:
If the target file does not exist, it will be created, then the string is
appended (effectively written as the first content).
If the target file already exists, CREATE is ignored and the string is
appended.
With this fix, users will no more get errors like
java.nio.file.NoSuchFileException for stderr when submitting jobs to standalone
cluster.
### Does this PR introduce _any_ user-facing change?
No, just tiny bug fix.
### How was this patch tested?
Tested manually.
### Was this patch authored or co-authored using generative AI tooling?
Closes #52789 from EricGao888/fix-writeString-in-standalone-driverRunner.
Authored-by: EricGao888 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
index 28de0f45eb27..5f2fe0f1ccbe 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
@@ -215,7 +215,7 @@ private[deploy] class DriverRunner(
val redactedCommand = Utils.redactCommandLineArgs(conf,
builder.command.asScala.toSeq)
.mkString("\"", "\" \"", "\"")
val header = "Launch Command: %s\n%s\n\n".format(redactedCommand,
"=".repeat(40))
- Files.writeString(stderr.toPath, header, StandardOpenOption.APPEND)
+ Files.writeString(stderr.toPath, header, StandardOpenOption.CREATE,
StandardOpenOption.APPEND)
CommandUtils.redirectStream(process.getErrorStream, stderr)
}
runCommandWithRetry(ProcessBuilderLike(builder), initialize, supervise)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
