This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 69312fa [SPARK-56701] Use Apache Spark 4.0.2 in `pi-with-comet`
example
69312fa is described below
commit 69312fa6be97d62af635791d22779481fc5474a6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri May 1 13:27:49 2026 -0700
[SPARK-56701] Use Apache Spark 4.0.2 in `pi-with-comet` example
### What changes were proposed in this pull request?
This PR upgrades the `pi-with-comet` example to Apache Spark 4.0.2.
### Why are the changes needed?
**GOAL**
- To align with the other examples which already use Apache Spark 4.0.2.
- To support and help the adoption of `Apache DataFusion Comet` with Spark
4.x more easily.
Note that `Apache DataFusion Comet` v0.15 has an experimental Spark 4.0
support. The upcoming `v0.16` will support `Apache Spark 4.0.2` officially.
- https://datafusion.apache.org/comet/user-guide/0.15/installation.html
-
https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html
<img width="938" height="335" alt="Screenshot 2026-05-01 at 13 12 21"
src="https://github.com/user-attachments/assets/5990d7dc-0ad2-4843-b732-b4b8cadaf1b8";
/>
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)
Closes #649 from dongjoon-hyun/SPARK-56701.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/pi-with-comet.yaml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/examples/pi-with-comet.yaml b/examples/pi-with-comet.yaml
index 3c03e06..2798720 100644
--- a/examples/pi-with-comet.yaml
+++ b/examples/pi-with-comet.yaml
@@ -30,7 +30,7 @@ spec:
spark.dynamicAllocation.shuffleTracking.enabled: "true"
spark.executor.extraClassPath: "local:///comet/comet.jar"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:3.5.8-java17"
+ spark.kubernetes.container.image: "apache/spark:4.0.2-scala"
spark.memory.offHeap.enabled: "true"
spark.memory.offHeap.size: "1g"
spark.plugins: "org.apache.spark.CometPlugin"
@@ -46,7 +46,7 @@ spec:
command: ["sh", "-c"]
args:
- |
- wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/0.15.0/comet-spark-spark3.5_2.12-0.15.0.jar";
+ wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/0.15.0/comet-spark-spark4.0_2.13-0.15.0.jar";
volumeMounts:
- name: comet
mountPath: /comet
@@ -68,7 +68,7 @@ spec:
command: ["sh", "-c"]
args:
- |
- wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/0.15.0/comet-spark-spark3.5_2.12-0.15.0.jar";
+ wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/0.15.0/comet-spark-spark4.0_2.13-0.15.0.jar";
volumeMounts:
- name: comet
mountPath: /comet
@@ -82,4 +82,4 @@ spec:
emptyDir:
sizeLimit: 200Mi
runtimeVersions:
- sparkVersion: "3.5.8"
+ sparkVersion: "4.0.2"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
