andygrove opened a new issue, #6645:
URL: https://github.com/apache/datafusion-comet/issues/6645
### Describe the bug
Spark's `DynamicPartitionPruningV1SuiteAEOn` test "SPARK-34637: DPP side
broadcast query stage is created firstly" fails in the Spark 3.4 `sql_core-1`
shard:
```
hasReuse was false SubqueryBroadcast dynamicpruning#66615, 0,
[store_id#66614], [id=#157769]
...
should have been reused in
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
*(4) BroadcastHashJoin [store_id#65490], [store_id#66614], Inner,
BuildLeft, false
```
The initial plan broadcasts the right side, which the DPP subquery reuses.
The final plan broadcasts the left side instead, so the subquery's broadcast is
never reused.
It fails on #4565 (run 37212552551) and on #5841 (run 37222066341). #5841
changes only CI, and both runs merged with `main` after #6547. It passed on
#6415's run on 2026-09-29 (`main` at e3d52feaf6). Spark 3.4 SQL tests run only
on pull requests that carry `run-spark-3.4-tests`, so neither the merge queue
nor the nightly catches it.
### Cause
#6547 patches the stale `Scan` leaf of a native operator that AQE reuses
over a shuffle stage (the final aggregate of a two-phase aggregate). It does
this in `CometExecRule` while that rule runs as AQE's query stage preparation
rule, so during each re-plan. The re-planned plan then differs from the current
one, and AQE adopts it. After that, AQE cannot link the broadcast stage it
later builds over the reused operator back into its logical plan, because that
operator's logical node now sits inside a `LogicalQueryStage`. The join's build
side is no longer pinned.
Once the probe side's shuffle stage materializes, the next re-plan picks the
smaller side using runtime statistics. On Spark 3.4 the AQE DPP probe side
stays in Spark, and its shuffle reports a smaller size than Comet's, so the
join flips to `BuildLeft`.
### Steps to reproduce
Run the test against Spark 3.4 with Comet enabled. It passes with
`spark.comet.shuffle.directRead.enabled=false`, which keeps #6547's refresh
from firing, and fails with the default.
### Expected behavior
AQE re-plans should not change because of Comet's shuffle read refresh, and
the DPP broadcast should be reused as it was before #6547.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
