sivakumar-mahalingam opened a new pull request, #4392:
URL: https://github.com/apache/amoro/pull/4392
## Why are the changes needed?
The external Spark optimizer currently follows Amoro's global Spark and
Scala versions. This prevents the optimizer from running with Spark 4.1 images
because Spark 4.1 uses Scala 2.13 and requires a newer Java and dependency
baseline.
This pull request adds Spark 4.1 support specifically for the external Spark
optimizer used with native Iceberg tables. It does not enable general Spark 4.1
support for AMS, mixed-format Spark modules, or Paimon.
Closes #4386.
## Brief change log
- Add optimizer-specific Spark version, Spark major version, Scala binary
version, and Jackson properties.
- Add the `optimizer-spark-4.1` Maven profile targeting Spark 4.1.3, Scala
2.13, and JDK 17.
- Preserve the existing Spark 3.5 optimizer build and global Spark
configuration.
- Produce a distinct `amoro-optimizer-spark-4.1_2.13` artifact.
- Keep Spark, Scala, and unshaded Jackson classes out of the shaded
optimizer JAR.
- Update optimizer distribution assembly and Docker build logic to select
the optimizer-specific artifact.
- Add a Spark 4.1 optimizer Docker matrix entry while retaining Spark 3.5 as
the default image.
- Add a JDK 17 and Spark 4.1 CI job with a shaded-JAR isolation check.
- Expand the local-mode optimizer test suite to two positive and three
negative cases covering:
- Successful local Spark execution and optimizer output.
- Factory property and native-Iceberg metadata propagation.
- Worker-side optimizer failure propagation.
- Missing executor-factory metadata.
- Malformed serialized task input.
- Document the optimizer-only Spark 4.1 build command in the README.
## How was this patch tested?
- [x] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
### Test Execution
Spark 4.1.3 optimizer tests on JDK 17:
```powershell
.\mvnw.cmd test -pl amoro-optimizer/amoro-optimizer-spark -am
'-Pjava17,optimizer-spark-4.1' '-Dtest=TestSparkOptimizerLocalMode'
'-Djacoco.skip=true' -B -ntp
```
Result:
```powershell
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
Spark 3.5.8 regression tests on JDK 11:
```powershell
.\mvnw.cmd clean test -pl amoro-optimizer/amoro-optimizer-spark -am
'-Pjava11,spark-3.5' '-Dtest=TestSparkOptimizerLocalMode' '-Djacoco.skip=true'
-B -ntp
```
Result:
```powershell
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
Native-Iceberg physical rewrite regression suite on JDK 17 with the Spark
4.1 optimizer profile:
```powershell
.\mvnw.cmd test -pl amoro-format-iceberg -am '-Pjava17,optimizer-spark-4.1'
'-Dtest=IcebergRewriteExecutorTest' '-Djacoco.skip=true' -B -ntp
```
Result:
```powershell
Tests run: 54, Failures: 0, Errors: 0, Skipped: 20
BUILD SUCCESS
```
Additional validation:
- Inspected the shaded Spark 4.1 optimizer artifact and confirmed that it
contains no Spark, Scala, or unshaded Jackson classes.
- Built the optimizer image using the official `apache/spark:4.1.3` base
image.
- Ran the packaged `SparkOptimizer` entry point through Spark 4.1.3 local
mode. It initialized and reached AMS registration; registration then failed as
expected because no AMS instance was running.
- Ran a native Iceberg v2 rewrite inside the Spark 4.1 image using the
production `SparkOptimizerExecutor` and `IcebergRewriteExecutorFactory`. Two
Parquet input files were rewritten into one output file containing both records.
- Submitted the optimizer using Spark 4.1.3 Kubernetes cluster mode on a
temporary kind cluster. The driver and executor started and reached AMS
registration without Spark, Scala, or dependency linkage errors.
- Ran repository formatting checks and `git diff --check`.
A full `clean verify` was attempted but stopped during Apache RAT validation
because of the existing tracked test fixture:
```powershell
amoro-format-mixed/amoro-mixed-hive/src/test/resources/string_is_bytes.parquet
```
Targeted Spark optimizer and native-Iceberg test suites completed
successfully.
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? README
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
