jeffw13 opened a new pull request, #5821: URL: https://github.com/apache/datafusion-comet/pull/5821
## Which issue does this PR close? Closes #5819. ## Rationale for this change When Spark 4 AQE replaces an empty query stage with `EmptyRelationExec`, Comet does not recognize the new leaf as a native input. Supported joins and aggregates above it can fall back to Spark. ## What changes are included in this PR? - Add `CometEmptyRelationExec` through the existing Arrow-input path, preserving Spark's output attributes and zero partitions without executing the eliminated subtree. - Register the operator through Spark version shims and enable it by default with `spark.comet.exec.emptyRelation.enabled`. - Preserve existing type, join and aggregate-buffer fallback rules. - Keep Spark's Parquet writer for inputs containing the new empty operator so empty writes still produce a readable file with schema metadata. This covers the newly supported input and leaves the broader issue #5303 open. - Document the operator and its writer fallback. ## How are these changes tested? Eight regression tests cover attributes, canonicalization, configuration, zero partitions and an empty input stream, global/grouped COUNT and SUM, 33 empty-side hash-join combinations, aggregate-buffer and existence-join fallback, reused broadcasts, AQE, and empty Parquet overwrite/readback. Local validation used the upstream Maven profiles and unchanged native dependencies on Linux amd64 / OpenJDK 17: | Spark / Scala | Validation | Result | | --- | --- | --- | | 4.0.4 / 2.13 | New empty-input tests and full conversion-rule, aggregate and Parquet-writer suites | 158 passed | | 4.1.3 / 2.13 | Same suites | 160 passed | | 3.4.3 / 2.12 | Compilation, conversion-rule and Parquet-writer suites | 59 passed | | 3.5.9 / 2.13 | Same compatibility checks | 61 passed | Spark 4.0 had two version-specific cancellations. Spark 3.4/3.5 had 11/9 version-specific cancellations, including the eight Spark 4-only tests. The Spark 4 aggregate suite retains its two existing ignored tests. Also passed: native build, 129 unchanged Spark 4.0 TPC-DS plan snapshots, Maven packaging, Scalafix, Scalastyle, Spotless and Markdown formatting. Removing only the writer guard makes the new overwrite/readback test fail with `PATH_NOT_FOUND`; restoring it passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
