kalluripradeep opened a new pull request, #64391:
URL: https://github.com/apache/airflow/pull/64391
## Summary
Closes #50958
When running Spark jobs on Kubernetes with a service mesh like Istio,
the sidecar proxy container does not automatically exit when the Spark
driver finishes. This leaves the pod in a running state indefinitely,
blocking job completion in Airflow.
This PR adds a `post_submit_commands` parameter to `SparkSubmitHook`
that runs a list of shell commands after the Spark job finishes —
whether it completes successfully or is killed via `on_kill()`.
**Example usage:**
```python
SparkSubmitHook(
application="my_job.py",
post_submit_commands=[
"curl -sf -X POST http://localhost:15020/quitquitquit";
],
)
```
## Changes
- Added `post_submit_commands: list[str] | None = None` parameter to
`SparkSubmitHook.__init__()`
- Added `_run_post_submit_commands()` method that runs each command via
`subprocess.run(shell=True)` with a 30s timeout, logging output and
non-zero exit codes as warnings (never raises)
- Called from both `submit()` (after job completion) and `on_kill()`
(after process termination)
- Added tests covering success, failure resilience, timeout, and
backward compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
