jeffw13 opened a new issue, #5819:
URL: https://github.com/apache/datafusion-comet/issues/5819
### What is the problem the feature request solves?
Spark 4.x can replace a query stage that produces no rows with
`EmptyRelationExec` during adaptive execution. Comet does not currently
recognize this operator as a native input, so supported operators above it can
fall back to Spark.
For example, with AQE enabled, an aggregate over an empty grouped result can
acquire an `EmptyRelationExec` input after the shuffle completes:
```sql
SELECT count(*), sum(v)
FROM (
SELECT id % 2 AS k, sum(id) AS v
FROM range(0, 10, 1, 2)
WHERE id < 0
GROUP BY id % 2
)
```
The result must remain one row containing `0` and `NULL`. A grouped
aggregate over the same empty input must produce no rows.
### Describe the potential solution
Add a columnar `CometEmptyRelationExec` using the existing native
Arrow-input path. Preserve the output attributes and zero partitions of Spark's
operator, without executing the eliminated logical subtree. Register it on
Spark 4.x through a version shim and add an operator enable/disable setting.
Test both the results and the executed plans for aggregates, supported
joins, AQE and reused exchanges. Preserve existing type and operator fallback
rules.
Parquet writes need an additional guard: admitting this zero-partition input
must not route a write through the native writer when that would omit the
schema-bearing empty file. Keep Spark's writer for those inputs and add an
overwrite/readback regression test.
### Additional context
Related to #5303, which tracks native Parquet writes producing no output
file for a zero-partition RDD. The writer fallback here would protect the newly
supported input; it would not resolve all cases in that issue.
This adds support for consuming Spark's empty relation. It does not change
Spark's rules for discovering or propagating empty relations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
