mbutrovich commented on issue #3510:
URL:
https://github.com/apache/datafusion-comet/issues/3510#issuecomment-4338886241
#### CometPlanAdaptiveDynamicPruningFilters simplification after makeCopy fix
`CometPlanAdaptiveDynamicPruningFilters.apply()` currently has two cases in
`transformUp`: one for `CometNativeScanExec` (uses explicit `copy()` + manual
dual-filter fixup) and one for non-Comet nodes like V2 `BatchScanExec` (uses
`transformExpressions`). Ideally this would be a single
`plan.transformAllExpressions` call, matching Spark's
`PlanAdaptiveDynamicPruningFilters`.
We can't do that today because `transformAllExpressions` calls
`mapExpressions`, which calls `makeCopy` when expressions change.
`CometNativeScanExec.makeCopy` loses the `@transient scan: CometScanExec` field
and any prior expression transformations. The explicit `copy()` path works
around this, and the dual-filter fixup (converting
`CometScanExec.partitionFilters` separately) exists because the inner scan is
`@transient` and invisible to expression tree traversals.
Once the makeCopy issue is fixed or the CometScanExec wrapping is removed
(both discussed in the design debt comment above), `apply()` collapses to:
```scala
override def apply(plan: SparkPlan): SparkPlan = {
if (!conf.dynamicPartitionPruningEnabled) return plan
plan.transformAllExpressions {
case dpe @ DynamicPruningExpression(inSub: InSubqueryExec) =>
extractSABData(inSub) match {
case Some(sabData) => convertSAB(inSub, sabData, plan)
case None => dpe
}
}
}
```
No separate cases, no `convertNativeScanDPP`, no `convertNonCometNodeDPP`,
no `hasWrappedSAB`, no `hasCometSAB`, and no dual-filter fixup (because there
would be no separate inner scan with its own partition filters).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
