adriangb opened a new issue, #26238: URL: https://github.com/apache/datafusion/issues/26238
### Is your feature request related to a problem or challenge? Follow-up to https://github.com/apache/datafusion/pull/25580 (`WITH ... AS MATERIALIZED`), part of https://github.com/apache/datafusion/issues/17737. That PR adds the logical nodes `MaterializedCte` and `MaterializedCteScan` as built-in `UserDefinedLogicalNode`s inside `LogicalPlan::Extension`. This kept the PR small, because a new `LogicalPlan` variant must be handled in every `match` on `LogicalPlan`. It was open question 1 of that PR. The cost is that rules that must know about the node find it by a downcast: - `PushDownFilter` has a special case that moves a filter onto the continuation and never into the shared body. - `PullUpCorrelatedExpr` (`decorrelate.rs`) rewrites only the continuation. - `LogicalPlan::head_output_expr` forwards to the continuation, so that `IN (subquery)` can be planned. - The physical planner finds both nodes by a downcast. The generic handling of extension nodes is conservative, so each rule that has no special case treats the node as a barrier. Code outside DataFusion (other optimizer rules, the unparser, Substrait and protobuf producers) sees an unknown extension node, and cannot see that the node has two inputs with different roles. ### Describe the solution you'd like Add first-class variants, in the same style as `LogicalPlan::RecursiveQuery`: - `LogicalPlan::MaterializedCte { id, name, cte, continuation }` - a leaf for the reference (a new variant, or a `TableScan` over a dedicated `TableSource`, as the work table of a recursive CTE does). Each rule then states what it does with the node in its `match`, and the downcasts are removed. This is an API change: a `match` on `LogicalPlan` in downstream code needs a new arm. It is easier to do this before more rules are written against the extension node (automatic materialization, filter pushdown into the body, column pruning). ### Describe alternatives you've considered Keep the extension nodes and add special cases as needed. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
