This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 4563830 [SPARK-53852] Add
`spark.kubernetes.driver.pod.excludedFeatureSteps` usage example
4563830 is described below
commit 45638303f2e54734004db469a320ac7dd570caee
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 8 23:59:28 2025 -0700
[SPARK-53852] Add `spark.kubernetes.driver.pod.excludedFeatureSteps` usage
example
### What changes were proposed in this pull request?
This PR aims to add `spark.kubernetes.driver.pod.excludedFeatureSteps`
usage example.
### Why are the changes needed?
Since `Apache Spark K8s Operator v0.6.0`, we can use new features from
`Apache Spark 4.1.0-preview2`.
- #364
For example, `spark.kubernetes.driver.pod.excludedFeatureSteps` allows us
to skip the specific feature steps. If we set the following, we can skip
`KerberosConfDriverFeatureStep` which fails always on Java 25 environment
currently.
- https://github.com/apache/spark/pull/51522
```
spark.kubernetes.driver.pod.excludedFeatureSteps:
"org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
```
**BEFORE**
```
25/10/09 04:22:23 INFO pi-preview default o.a.s.i.Logging You have not
specified a krb5.conf file locally or via a ConfigMap. Make sure that you have
the krb5.conf locally on the driver image.
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by
org.apache.hadoop.util.NativeCodeLoader in an unnamed module
(file:/opt/spark-operator/operator/spark-kubernetes-operator.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for
callers in this module
WARNING: Restricted methods will be blocked in a future release unless
native access is enabled
25/10/09 04:22:23 WARN pi-preview default o.a.h.u.NativeCodeLoader Unable
to load native-hadoop library for your platform... using builtin-java classes
where applicable
25/10/09 04:22:23 WARN pi-preview default o.a.s.i.Logging Fail to get
credentials
java.lang.UnsupportedOperationException: getSubject is not supported
at
java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:588)
at
org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep.liftedTree1$1(KerberosConfDriverFeatureStep.scala:95)
```
**AFTER**
```
25/10/09 04:20:34 INFO pi-preview default o.a.s.i.Logging You have not
specified a krb5.conf file locally or via a ConfigMap. Make sure that you have
the krb5.conf locally on the driver image.
```
### Does this PR introduce _any_ user-facing change?
No. This is an example update.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #379 from dongjoon-hyun/SPARK-53852.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/pi-preview.yaml | 1 +
examples/pi.yaml | 1 +
2 files changed, 2 insertions(+)
diff --git a/examples/pi-preview.yaml b/examples/pi-preview.yaml
index 24fcded..339ba1e 100644
--- a/examples/pi-preview.yaml
+++ b/examples/pi-preview.yaml
@@ -25,6 +25,7 @@ spec:
spark.dynamicAllocation.maxExecutors: "3"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
spark.kubernetes.container.image:
"apache/spark:4.1.0-preview2-java21-scala"
+ spark.kubernetes.driver.pod.excludedFeatureSteps:
"org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
diff --git a/examples/pi.yaml b/examples/pi.yaml
index 1549622..abfb530 100644
--- a/examples/pi.yaml
+++ b/examples/pi.yaml
@@ -25,6 +25,7 @@ spec:
spark.dynamicAllocation.maxExecutors: "3"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
spark.kubernetes.container.image: "apache/spark:4.0.1"
+ spark.kubernetes.driver.pod.excludedFeatureSteps:
"org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
