manuzhang opened a new pull request, #5358: URL: https://github.com/apache/datafusion-comet/pull/5358
## Which issue does this PR close? Closes #4329. ## Rationale for this change Spark 3.4 was deprecated in Comet 1.0.0 and is scheduled for removal in 1.1.0. Dropping it reduces the supported build/test matrix and removes compatibility paths that are no longer needed now that Spark 3.5 is the oldest supported release. ## What changes are included in this PR? - Remove the Spark 3.4 Maven profile, source/test shims, Spark SQL patch, CI jobs, release artifacts, generated compatibility page, and documentation references. - Simplify shared code and tests to assume Spark 3.5 or newer, including removing the Spark 3.4 AQE dynamic partition-pruning fallback. - Promote the Spark 3.5 plan-stability overrides into the base golden directories and keep Spark 4.x fallback chains intact. - Retarget the Iceberg 1.8 compatibility job to Spark 3.5.9 and JDK 17. ## How are these changes tested? - `make format PROFILES=-Pspark-3.5` - `./mvnw clean test-compile -DskipTests -Pspark-4.0` - `./mvnw clean test-compile -DskipTests -Pspark-4.1` - `./mvnw clean test-compile -DskipTests -Pspark-4.2` - Focused Spark 3.5 runtime coverage for AQE DPP and hash joins - `CometTPCDSV1_4_PlanStabilitySuite` on Spark 3.5 (97 tests) - `CometTPCDSV2_7_PlanStabilitySuite` on Spark 3.5 (32 tests) - YAML, shell syntax, Python bytecode, active-reference, and `git diff --check` validation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
