dwsmith1983 opened a new issue, #6901: URL: https://github.com/apache/datafusion-comet/issues/6901
### Describe the bug `DPP - single-split sort-merge join (AQE=true, pruned=a)` and `(AQE=true, pruned=b)` in `CometIcebergNativeSuite`, added in #6814, fail on the Spark 3.5 profile: ``` List() was empty Expected AQE to replace the pruned join with an empty relation ``` Found in #5867: https://github.com/apache/datafusion-comet/actions/runs/38276011652/job/114910128309. #6814's own Spark 3.5 `[scans]` job failed on the same two tests (https://github.com/apache/datafusion-comet/actions/runs/38066634294/job/114798528903), and they fail the same way on main (38f394e27). Spark 3.4, 4.0, 4.1 and 4.2 pass. The pull request tier runs only Spark 4.1, so main's CI doesn't run this profile. The test asserts a `CometEmptyRelationExec` whenever `isSpark35Plus && aqeEnabled && scans.isEmpty`. On Spark 3.5, `AQEPropagateEmptyRelation` replaces the pruned join with an empty `LocalRelation`, which plans as `LocalTableScanExec`. The `EmptyRelation` node, and with it `EmptyRelationExec`, only arrived in Spark 4.0 (SPARK-48466), and `ShimCometEmptyRelation` returns `None` on 3.x. `spark.comet.exec.localTableScan.enabled` defaults to false, so the empty scan and the aggregates above it stay in Spark: ``` HashAggregate +- ShuffleQueryStage +- Exchange SinglePartition +- HashAggregate +- LocalTableScan <empty> ``` The query result is correct. Only the plan check is wrong for 3.5. ### Steps to reproduce ```shell ./mvnw test -Pspark-3.5 -Pscala-2.13 -Dtest=none \ -Dsuites='org.apache.comet.CometIcebergNativeSuite DPP - single-split sort-merge join' ``` ### Expected behavior The test passes on every Spark profile Comet builds. ### Additional context Requiring `CometEmptyRelationExec` only on Spark 4.0+ and accepting the empty `LocalTableScanExec` on 3.5 would match what each version plans. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
