dwsmith1983 commented on issue #6264: URL: https://github.com/apache/datafusion-comet/issues/6264#issuecomment-5889137354
> does the #6264 fix cover an unconverted SubqueryAdaptiveBroadcastExec dropped outside canonicalization, or should I open a separate issue? It does not cover it, so a separate issue please. #6270 only changes the canonical copy of the scan that AQE compares for stage reuse. The filters the scan actually runs with never come from that copy. My suspect is `CometPlanAdaptiveDynamicPruningFilters`. When it looks for the broadcast to reuse, it searches the current stage and then the main query's plan. Spark's `PlanAdaptiveDynamicPruningFilters` searches the adaptive plan that owns the filter. In q23 the `store_sales` and `customer` join is inside the scalar subquery, which has its own adaptive plan, so the `date_dim` broadcast may never be found and the filter becomes `dynamicpruningexpression(true)`, the same result as Spark's no-reuse path. I have traced this in the source but not run it yet. Two things would confirm it: the `PartitionFilters` line of that `store_sales` scan in the final plan with `spark.comet.enabled=false` at the same threshold, and the Spark version. I will write a repro from your SF1 settings once the issue is open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
