andygrove opened a new issue, #7422:
URL: https://github.com/apache/datafusion-comet/issues/7422
## Describe the bug
On Spark 3.4, a native Iceberg scan with an AQE dynamic partition pruning
(DPP) filter fails at execution when an aggregate over the scan is the build
side of the DPP broadcast. `CometIcebergNativeSuite` "AQE DPP - V2 broadcast
query stage creation order (SPARK-34637)" reproduces it deterministically with
`-Pspark-3.4`:
```
org.apache.comet.CometNativeException: General execution error with reason:
Column index 1 is out of bound. Schema: .
at org.apache.comet.Native.executePlan(Native Method)
...
at
org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.$anonfun$relationFuture$1
```
The query is the SPARK-34637 shape: a CTE `SELECT store_id FROM fact WHERE
units_sold = 70 GROUP BY store_id`, self-joined, on an Iceberg table
partitioned by `store_id`. Spark 3.5 and later pass.
## Findings
The failing native plan is the final `CometHashAggregate` of the DPP build
side. Its JVM children are now `AQEShuffleRead` over a `ShuffleQueryStage`
(`CometExchange ... ENSURE_REQUIREMENTS`), but its serialized native plan still
has the partial aggregate, project, filter and the `IcebergScan` inlined
beneath it, as if no exchange existed. `findAllPlanData` therefore never
reaches a `CometIcebergNativeScanExec`, nothing injects the scan's common and
task data, and the native planner builds an `IcebergScan` with an empty
`required_schema` and no tasks. The filter's `units_sold@1` then fails to bind
against the empty schema.
The initial plan of that build side has no exchange between the two
aggregates, and AQE inserts one later, after Comet has already serialized the
final aggregate. Not yet established: why the initial plan needs no exchange.
Spark 3.4's `BatchScanExec` may report a different output partitioning than
`CometIcebergNativeScanExec`, which reports `UnknownPartitioning(0)` since
#6268.
## Steps to reproduce
```sh
make
./mvnw test -Pspark-3.4 -Dtest=none \
-Dsuites="org.apache.comet.CometIcebergNativeSuite AQE DPP - V2 broadcast
query stage creation order"
```
## Expected behavior
The query returns the same rows as Spark.
## Additional context
Found while running the Spark 3.4 jobs under `run-all-spark-profiles` on
#7402. The default-profile pull request tier does not run Spark 3.4, so this
was not caught earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
