This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 166df8b1df59 [SPARK-46344][CORE] Warn properly when a driver exists
successfully but master is disconnected
166df8b1df59 is described below
commit 166df8b1df5965fe3f15fecd5574545746b0b18f
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Dec 9 15:33:48 2023 -0800
[SPARK-46344][CORE] Warn properly when a driver exists successfully but
master is disconnected
### What changes were proposed in this pull request?
This PR aims to warn properly when a driver exists successfully but master
is disconnected.
### Why are the changes needed?
In this case, `Master` considers them `Error` eventually.
**Worker Log**
```
23/12/09 15:13:21 INFO Worker: Driver driver-20231209151301-0003 exited
successfully
=== Master is disconnected here ===
23/12/09 15:13:53 WARN Worker: Driver driver-20231209151332-0004 exited
successfully while master is disconnected.
=== A new master starts and is connected here ===
23/12/09 15:17:10 INFO Worker: Driver driver-20231209151707-0000 exited
successfully
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44278 from dongjoon-hyun/SPARK-46344.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
index 1422a1484f8d..785129e1d818 100755
--- a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
@@ -878,7 +878,12 @@ private[deploy] class Worker(
case DriverState.FAILED =>
logWarning(s"Driver $driverId exited with failure")
case DriverState.FINISHED =>
- logInfo(s"Driver $driverId exited successfully")
+ registrationRetryTimer match {
+ case Some(_) =>
+ logWarning(s"Driver $driverId exited successfully while master is
disconnected.")
+ case _ =>
+ logInfo(s"Driver $driverId exited successfully")
+ }
case DriverState.KILLED =>
logInfo(s"Driver $driverId was killed by user")
case _ =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
