This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 1a6cb2c [SPARK-56849] Use Spark 4.1.1 in `Apache DataFusion Comet`
example
1a6cb2c is described below
commit 1a6cb2c50fe662023601feb11a08a11e048ecea2
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed May 13 10:22:36 2026 -0700
[SPARK-56849] Use Spark 4.1.1 in `Apache DataFusion Comet` example
### What changes were proposed in this pull request?
This PR aims to make `tests/e2e/pi-with-comet` use Apache Spark 4.1.1 by
updating `examples/pi-with-comet.yaml`.
### Why are the changes needed?
To test the latest Apache Spark release. Recently, `Apache DataFusion
Comet` 0.16 starts to support Apache Spark 4.1.x with a known limitation.
-
https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html#spark-4-1
<img width="946" height="288" alt="Screenshot 2026-05-13 at 09 33 13"
src="https://github.com/user-attachments/assets/84c689d3-2062-42d7-8b48-365690101ec8";
/>
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7
Closes #674 from dongjoon-hyun/SPARK-56849.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/pi-with-comet.yaml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/examples/pi-with-comet.yaml b/examples/pi-with-comet.yaml
index 78fc91d..9113581 100644
--- a/examples/pi-with-comet.yaml
+++ b/examples/pi-with-comet.yaml
@@ -30,7 +30,7 @@ spec:
spark.dynamicAllocation.shuffleTracking.enabled: "true"
spark.executor.extraClassPath: "local:///comet/comet.jar"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:4.0.2-scala"
+ spark.kubernetes.container.image: "apache/spark:4.1.1-scala"
spark.memory.offHeap.enabled: "true"
spark.memory.offHeap.size: "1g"
spark.plugins: "org.apache.spark.CometPlugin"
@@ -46,7 +46,7 @@ spec:
command: ["sh", "-c"]
args:
- |
- wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/0.16.0/comet-spark-spark4.0_2.13-0.16.0.jar";
+ wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.1_2.13/0.16.0/comet-spark-spark4.1_2.13-0.16.0.jar";
volumeMounts:
- name: comet
mountPath: /comet
@@ -68,7 +68,7 @@ spec:
command: ["sh", "-c"]
args:
- |
- wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/0.16.0/comet-spark-spark4.0_2.13-0.16.0.jar";
+ wget -O /comet/comet.jar
"https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.1_2.13/0.16.0/comet-spark-spark4.1_2.13-0.16.0.jar";
volumeMounts:
- name: comet
mountPath: /comet
@@ -82,4 +82,4 @@ spec:
emptyDir:
sizeLimit: 200Mi
runtimeVersions:
- sparkVersion: "4.0.2"
+ sparkVersion: "4.1.1"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
