james-willis opened a new issue, #4059: URL: https://github.com/apache/datafusion-comet/issues/4059
### What is the problem the feature request solves? The [installation docs](https://datafusion.apache.org/comet/user-guide/installation.html) list Comet as supporting Java 11/17 only, and the Spark 4.0.1 (experimental) row is Java 17 only. This is a blocker for downstream consumers who want to run Comet alongside Spark 4 on a Java 21 runtime. Spark 4.x officially supports Java 17 and 21 ([ref](https://spark.apache.org/docs/4.0.1/)). Java 21 is an LTS release, ~2.5 years old at this point, and the broader Spark ecosystem (Arrow, Netty, Iceberg, Delta, etc.) has shipped Java 21 compatibility. Comet is currently one of the last pieces of the stack with no official Java 21 validation. ### Describe the potential solution Add a Java 21 row to the `pr_build_linux.yml` matrix for Spark 4.0.x (to start). Concretely: - Add `Spark 4.0, JDK 21` to the build matrix in `.github/workflows/pr_build_linux.yml` and `spark_sql_test.yml`. - Extend the conditional `JAVA_TOOL_OPTIONS` logic so the Java 17 `--add-opens`/`--add-exports` flags (which remain valid) are also applied on Java 21. Spark 4's own launcher scripts already enumerate the right set of flags for both JDK versions, so most of the work is copying that prior art. - Once CI is green, update the compatibility matrix in `docs/source/user-guide/latest/installation.md` to list Java 11/17/21 alongside Spark 4.0 (and the 3.5.x rows if CI extends there too). Incremental approach if a full matrix expansion is too much: start with just `Spark 4.0 + Java 21` since that's the Spark version most likely to be paired with Java 21 in production. ### Additional context We maintain a fork of Comet ([wherobots/dbv2](https://github.com/wherobots/dbv2)) and are about to land a Spark 4.0 / Java 21 CI profile internally. If that validation comes back clean, we'd be happy to contribute a PR upstream porting the change (matrix addition + doc update). Related upstream work: - [SPARK-43831 Build and Run Spark on Java 21](https://issues.apache.org/jira/browse/SPARK-43831) (resolved) - Java 21 build issue [#129](https://github.com/apache/datafusion-comet/issues/129) (closed 2024-02, Scala 2.12 compiler bug that's long since been fixed) No known open Java 21-specific bugs on the tracker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
