This is an automated email from the ASF dual-hosted git repository.
ueshin pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new d4754bace6e3 [SPARK-51608][PYTHON] Log exception on Python runner
termination
d4754bace6e3 is described below
commit d4754bace6e31729e1ba13eb1e59d89f3d928b49
Author: dmitry.sorokin <[email protected]>
AuthorDate: Fri Mar 28 13:13:20 2025 -0700
[SPARK-51608][PYTHON] Log exception on Python runner termination
### What changes were proposed in this pull request?
There are situations, when python worker may throw exceptions that are not
covered by any other kind of logging (for example custom
implementation/unexpected throw from existing code)
In these situations it is hard to debug which exactly issue happened, as
there is no stack trace in logs and no actual message from exception.
To address this issue exception is added for the logging call (rare
exception situation while stopping python worker)
### Why are the changes needed?
To effectively debug rare situations with worker termination.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
no need
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #49890 from antban/log-exception-on-worker-termination.
Lead-authored-by: dmitry.sorokin <[email protected]>
Co-authored-by: dsorokin <[email protected]>
Signed-off-by: Takuya Ueshin <[email protected]>
(cherry picked from commit 28a756222800810868994c1cd6de09a9bf3d6584)
Signed-off-by: Takuya Ueshin <[email protected]>
---
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
index 3ff78da6e584..eed63142fcc4 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
@@ -273,7 +273,7 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
worker.stop()
} catch {
case e: Exception =>
- logWarning("Failed to stop worker")
+ logWarning(log"Failed to stop worker", e)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
