andygrove opened a new issue, #4101:
URL: https://github.com/apache/datafusion-comet/issues/4101
## Describe the proposed change
The Iceberg CI workflow (`.github/workflows/iceberg_spark_test.yml`)
currently runs a fully-crossed matrix:
- 3 job types (`iceberg-spark`, `iceberg-spark-extensions`,
`iceberg-spark-runtime`)
- 3 Iceberg versions (1.8.1, 1.9.1, 1.10.0)
- 2 Spark versions (3.4.3, 3.5.8)
- 2 JDK versions (11, 17)
- 1 Scala version (2.13)
That's 36 jobs per PR. Adding Spark 4.0.1 / 4.1.1 to the same pattern would
push it higher.
The JDK dimension is the easiest to trim. Each Spark version effectively has
one preferred JDK in practice:
- Spark 3.4 → JDK 11
- Spark 3.5 → JDK 17 (and Spark 4.x will require 17+ anyway)
Pinning JDK per Spark version (using a matrix `include:` rather than full
cross-product) cuts the matrix in half: 36 → 18 jobs, removing 18 redundant
combinations.
## Rationale
- Iceberg's own CI does not exhaustively cross JDK with every Spark version;
we are overcovering relative to upstream.
- JDK 11 vs 17 differences that affect Comet are caught elsewhere:
`pr_build_linux.yml` and `spark_sql_test.yml` both run JDK 11 + 17 against
multiple Spark versions.
- The Iceberg suites are about Iceberg/Comet integration, not JVM-level
differences; the same JDK delta is unlikely to surface there but not in the
broader matrix.
- Frees CI capacity to add Spark 4.0.1 and 4.1.1 to the matrix without a net
increase in PR runtime.
## Proposed change
Replace the fully-crossed matrix in each of the three Iceberg jobs with an
`include:` list that pins JDK per Spark version, e.g.
```yaml
strategy:
matrix:
iceberg-version: [{short: '1.8', full: '1.8.1'}, {short: '1.9', full:
'1.9.1'}, {short: '1.10', full: '1.10.0'}]
include:
- spark-version: {short: '3.4', full: '3.4.3'}
java-version: 11
- spark-version: {short: '3.5', full: '3.5.8'}
java-version: 17
```
Net effect: 36 → 18 PR jobs.
## Additional context
Part of a broader CI cleanup ahead of adding Spark 4.0.1 and 4.1.1 to the
test matrix. Other potential follow-ups (separate issues): drop Iceberg 1.9
(boundary coverage only), reduce macOS matrix, tier nightly vs PR-blocking
tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
