This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 2f9e2a1 [SPARK-53827] Exclude `commons-(codec|compress)` and
`com.ibm.icu` transitive dependencies
2f9e2a1 is described below
commit 2f9e2a1219e95725538e56586b2430cfcc21c35f
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Oct 7 17:03:33 2025 -0700
[SPARK-53827] Exclude `commons-(codec|compress)` and `com.ibm.icu`
transitive dependencies
### What changes were proposed in this pull request?
This PR aims to exclude 3 transitive dependencies to reduce the size of
`spark-kubernetes-operator.jar` about **11% (from 149MB to 133MB)**.
- `commons-codec:*`
- `org.apache.commons:commons-compress`
- `com.ibm.icu:*`
### Why are the changes needed?
Apache Spark K8s Operator is not using these. We had better remove this to
reduce `spark-kubernetes-operator.jar` file size (and the Docker image size)
from 149MB to 133MB.
**BEFORE**
```
$ docker run -it --rm apache/spark-kubernetes-operator:0.6.0-SNAPSHOT ls
-alh /opt/spark-operator/operator | tail -n1
/
-rw-r--r-- 1 spark spark 149M Oct 7 23:21 spark-kubernetes-operator.jar
```
**AFTER**
```
$ docker run -it --rm apache/spark-kubernetes-operator:0.6.0-SNAPSHOT ls
-alh /opt/spark-operator/operator | tail -n1
/
-rw-r--r-- 1 spark spark 133M Oct 7 23:28 spark-kubernetes-operator.jar
```
### Does this PR introduce _any_ user-facing change?
No behavior change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #375 from dongjoon-hyun/SPARK-53827.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
config/checkstyle/checkstyle.xml | 2 ++
spark-operator/build.gradle | 3 +++
spark-submission-worker/build.gradle | 3 +++
3 files changed, 8 insertions(+)
diff --git a/config/checkstyle/checkstyle.xml b/config/checkstyle/checkstyle.xml
index 315a62a..9f02ee6 100644
--- a/config/checkstyle/checkstyle.xml
+++ b/config/checkstyle/checkstyle.xml
@@ -201,8 +201,10 @@
value="Please use the `assertThrows` method to test for
exceptions."/>
</module>
<module name="IllegalImport">
+ <property name="illegalPkgs" value="org.apache.commons.codec"/>
<property name="illegalPkgs" value="org.apache.commons.collections"/>
<property name="illegalPkgs" value="org.apache.commons.collections4" />
+ <property name="illegalPkgs" value="org.apache.commons.compress" />
<property name="illegalPkgs" value="org.apache.commons.lang"/>
<property name="illegalPkgs" value="org.apache.commons.lang3" />
<property name="illegalPkgs" value="org.apache.log4j"/>
diff --git a/spark-operator/build.gradle b/spark-operator/build.gradle
index a893257..b5029dc 100644
--- a/spark-operator/build.gradle
+++ b/spark-operator/build.gradle
@@ -40,9 +40,12 @@ dependencies {
implementation(libs.metrics.jvm)
compileOnly(libs.spark.core) {
exclude group: "com.github.luben"
+ exclude group: "com.ibm.icu"
+ exclude group: "commons-codec"
exclude group: "commons-collections", module: "commons-collections"
exclude group: "io.netty", module: "netty-tcnative-boringssl-static"
exclude group: "io.netty", module: "netty-tcnative-classes"
+ exclude group: "org.apache.commons", module: "commons-compress"
exclude group: "org.apache.logging.log4j"
exclude group: "org.fusesource.leveldbjni"
exclude group: "org.lz4"
diff --git a/spark-submission-worker/build.gradle
b/spark-submission-worker/build.gradle
index 69d0b9e..2e30145 100644
--- a/spark-submission-worker/build.gradle
+++ b/spark-submission-worker/build.gradle
@@ -26,10 +26,13 @@ dependencies {
implementation(libs.spark.kubernetes) {
exclude group: "com.github.luben"
+ exclude group: "com.ibm.icu"
+ exclude group: "commons-codec"
exclude group: "commons-collections", module: "commons-collections"
exclude group: "io.fabric8"
exclude group: "io.netty", module: "netty-tcnative-boringssl-static"
exclude group: "io.netty", module: "netty-tcnative-classes"
+ exclude group: "org.apache.commons", module: "commons-compress"
exclude group: "org.apache.logging.log4j"
exclude group: "org.fusesource.leveldbjni"
exclude group: "org.lz4"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
