This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 832e4d034827 [SPARK-49176][K8S][DOCS] Fix
`spark.ui.custom.executor.log.url` docs by adding K8s
832e4d034827 is described below
commit 832e4d0348277ee012aba6e4a7d31f1d9bc849c3
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Aug 9 16:33:41 2024 +0900
[SPARK-49176][K8S][DOCS] Fix `spark.ui.custom.executor.log.url` docs by
adding K8s
### What changes were proposed in this pull request?
This PR aims to fix `spark.ui.custom.executor.log.url` configuration's
documentation by adding K8s on top of YARN.
```
- For now, only YARN mode supports this configuration
+ For now, only YARN and K8s cluster manager supports this configuration
```
In addition, K8s example is added to `running-on-kubernetes.md` docs to
help users refer the existing environment variables, `SPARK_APPLICATION_ID` and
`SPARK_EXECUTOR_ID`, more easily to enable this feature.
-
https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config
### Why are the changes needed?
To fix a wrong description.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review. This is a doc-only PR.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47681 from dongjoon-hyun/SPARK-49176.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
docs/configuration.md | 2 +-
docs/running-on-kubernetes.md | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 6211400713d6..2881660eded6 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1666,7 +1666,7 @@ Apart from these, the following properties are also
available, and may be useful
which will be also effective when accessing the application on history
server. The new log urls must be
permanent, otherwise you might have dead link for executor log urls.
<p/>
- For now, only YARN mode supports this configuration
+ For now, only YARN and K8s cluster manager supports this configuration
</td>
<td>3.0.0</td>
</tr>
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 6e46dbbb858d..3fb963ea1eef 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -430,6 +430,14 @@ The same logs can also be accessed through the
[Kubernetes
dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/)
if installed on
the cluster.
+When there exists a log collection system, you can expose it at Spark Driver
`Executors` tab UI. For example,
+
+```
+spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_APP_ID='$(SPARK_APPLICATION_ID)'
+spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_EXECUTOR_ID='$(SPARK_EXECUTOR_ID)'
+spark.ui.custom.executor.log.url='https://log-server/log?appId={{APP_ID}}&execId={{EXECUTOR_ID}}'
+```
+
### Accessing Driver UI
The UI associated with any application can be accessed locally using
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
