adriangb opened a new issue, #26231: URL: https://github.com/apache/datafusion/issues/26231
### Is your feature request related to a problem or challenge? Follow-up to https://github.com/apache/datafusion/pull/25580 (`WITH ... AS MATERIALIZED`), part of https://github.com/apache/datafusion/issues/17737. A plan that contains a materialized CTE cannot be serialized with `datafusion-proto`. The new nodes have no protobuf encoding, so the default extension codec returns an error for them. Systems that send plans to other processes (for example Ballista or `datafusion-distributed`) cannot run a query that uses `WITH ... AS MATERIALIZED`. The nodes are: - logical: `MaterializedCte`, `MaterializedCteScan`; - physical: `MaterializedCteExec`, `MaterializedCteScanExec`. ### Describe the solution you'd like Add protobuf encodings for the four nodes. The physical nodes are not independent: each `MaterializedCteScanExec` shares one `MaterializedCteBuffer` with its `MaterializedCteExec`. The encoding should carry only the CTE id, and the decoder should bind the scans to the buffer again, as the planner does after physical planning. A scan inside a scalar subquery is not below its `MaterializedCteExec`, so the binding must be done on the whole decoded plan. A plan that is split so that the scans run in a different process from the body cannot share an in-memory buffer. That case can stay an error. ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
