pingzh opened a new pull request, #5537: URL: https://github.com/apache/datafusion-comet/pull/5537
## Which issue does this PR close? Part of #5352. This is the eighth PR in the Celeborn integration series and does not close the issue. Previous PRs: - 1/n: [#5473](https://github.com/apache/datafusion-comet/pull/5473) - 2/n: [#5476](https://github.com/apache/datafusion-comet/pull/5476) - 3/n: [#5481](https://github.com/apache/datafusion-comet/pull/5481) - 4/n: [#5491](https://github.com/apache/datafusion-comet/pull/5491) - 5/n: [#5501](https://github.com/apache/datafusion-comet/pull/5501) - 6/n: [#5513](https://github.com/apache/datafusion-comet/pull/5513) - 7/n: [#5531](https://github.com/apache/datafusion-comet/pull/5531) ## Rationale for this change The preceding PRs added the Celeborn-backed native shuffle writer, map-side lifecycle, and raw reader. This PR enables the planner to select that path through `CometCelebornShuffleManager` with explicit native opt-in. Unsupported exchanges must retain ordinary Spark/Celeborn shuffle without selecting Comet's JVM columnar shuffle. Fallback must also keep aggregate producers and consumers on compatible intermediate buffer formats. ## What changes are included in this PR? - Recognize the composite manager and require Comet execution, Comet shuffle, and explicit `spark.comet.shuffle.mode=native`. Default `auto` and `jvm` retain ordinary Spark/Celeborn shuffle. - Snapshot effective application configuration through Celeborn's configuration API, preserving defaults, aliases, and fallback-policy precedence. Reject native planning for I/O encryption, disabled stage reruns, unsupported configuration APIs, and applicable local-fallback policies or partition thresholds. Session overrides cannot bypass these application-level restrictions. - Require a native child and supported native partitioning/schema; preserve sticky AQE fallback and the existing local Comet shuffle behavior. Apply the shared guards to collect-limit and top-K shuffle producers as well. - Keep incompatible aggregate buffers on Spark when an exchange or intermediate merge cannot run natively. Restore affected native ancestors, preserve completed/stage boundaries, and retain safe mixed execution such as non-decimal AVG. - Make the shuffle-mode setting public, document configuration and limitations, and register the new planner suite in Linux and macOS CI. ## How are these changes tested? New planner and manager tests cover the opt-in matrix, runtime versus session configuration, supported and unsupported exchanges, AQE, special shuffle producers, fallback-policy boundaries, optional API failures, and immutable configuration snapshots. Executed Spark-reference comparisons cover fallback queries, collect-list/set buffer compatibility, and mixed DISTINCT/percentile/FIRST/LAST pipelines. Planner-only checks cover repeated rule application, native-ancestor rollback, and completed aggregate boundaries. The local Comet shuffle path has a matching aggregate regression. Local validation: - Final full-reactor Maven run on Spark 4.1.3 / Scala 2.13 / JDK 21: **116 tests passed**, including the corrected FIRST/LAST diagnostic assertion. - Broader 13-suite shuffle, JNI, configuration, and aggregate run: **379/380 passed** before the final diagnostic correction; its sole failure passed in the final rerun. - Rust core, shuffle, and JNI library tests: **312 passed**, with four existing HDFS tests ignored. - Spotless, ScalaStyle, `cargo fmt --all --check`, and workspace Clippy with warnings denied passed. - **63 configuration compatibility checks** passed against published Celeborn 0.6.3 and 0.7.0 clients. Scala 2.12 parsing and the corresponding reflected API signatures were also checked. Native plan-selection tests use a delegated test manager without executing remote native dependencies. No live Celeborn cluster or complete legacy Spark build matrix was run locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
