adriangb opened a new issue, #26233: URL: https://github.com/apache/datafusion/issues/26233
### Is your feature request related to a problem or challenge? Follow-up to https://github.com/apache/datafusion/pull/25580 (`WITH ... AS MATERIALIZED`), part of https://github.com/apache/datafusion/issues/17737. The buffered output of a materialized CTE (the in-memory batches, their memory reservation and the spill files) is released only when the plan and all replay streams are dropped. In a long query, the CTE is often read early, for example as the build side of a join, and the rest of the query then runs for a long time while the buffer still holds memory and disk space that nothing will read again. ### Describe the solution you'd like Release the buffer when the last partition of the last scan has finished or has been dropped. Points to solve: - The buffer knows how many scans are bound to it, and each scan knows its partition count, so it can count the partitions that are still to be read. - A scan partition that is never executed must not keep the buffer for ever, and must not get an empty result if it is executed late. The count must be of partitions that can still be executed, not only of streams that were started. - `reset_state` and a second execution of the same plan must still work. See https://github.com/apache/datafusion/issues/26229. A smaller step is to release each body partition when all scans have read it. ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
