andygrove opened a new pull request, #4119: URL: https://github.com/apache/datafusion-comet/pull/4119
## Which issue does this PR close? No tracking issue yet; opening this draft to start the conversation about Spark 4.2 support. ## Rationale for this change Spark 4.2.0-preview4 is published on Maven Central. Adding a build-only profile lets us start tracking 4.2 API drift incrementally rather than discovering everything in one pass when the GA release lands. This PR is stacked on top of #4118 (the spark-4.x shim consolidation). The first two commits are #4118; the third commit is the 4.2-specific work. ## What changes are included in this PR? A new `spark-4.2` Maven profile in `pom.xml` and `spark/pom.xml`. Profile properties: | property | value | | ------------------- | --------------- | | `spark.version` | `4.2.0-preview4`| | `scala.version` | `2.13.18` | | `parquet.version` | `1.17.0` | | `slf4j.version` | `2.0.17` | | `shims.majorVerSrc` | `spark-4.x` | | `shims.minorVerSrc` | `spark-4.2` | A new `spark/src/main/spark-4.2/` shim tree with `CometExprShim.scala` (copied unchanged from `spark-4.1`). Two Spark API changes between 4.1 and 4.2 required new shims. Because the new behavior diverges between 4.0/4.1 and 4.2, the new shim files live in the per-minor source roots (`spark-4.0`, `spark-4.1`, `spark-4.2`) rather than `spark-4.x`: - `DataSourceV2ScanExecBase.partitions` changed from `Seq[Seq[InputPartition]]` to `Seq[Option[InputPartition]]`. The `partitions` override on `CometBatchScanExec` was extracted into a new `ShimCometBatchScanExec` trait that exposes the value as `shimPartitions`; the concrete override in `CometBatchScanExec` simply delegates to it. - `DataSourceRDDPartition.inputPartitions: Seq[InputPartition]` was replaced by `inputPartition: Option[InputPartition]`. The reflective access in `CometIcebergNativeScan` now goes through a new `ShimDataSourceRDDPartition` helper that normalizes both shapes back to `Seq[InputPartition]`. Out of scope (intentionally not wired up): - Test sources, test profile dependencies (Iceberg runtime, Jetty) - Plan-stability golden files for 4.2 - CI matrix entries for 4.2 - Any 4.2-specific expression handling beyond what 4.1 already does ## How are these changes tested? - `./mvnw -Pspark-4.0 -DskipTests clean compile` succeeds. - `./mvnw -Pspark-4.1 -DskipTests clean compile` succeeds. - `./mvnw -Pspark-4.2 -DskipTests clean compile` succeeds. - No runtime tests are wired up for the new profile yet; that is follow-up work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
