adriangb commented on issue #17737: URL: https://github.com/apache/datafusion/issues/17737#issuecomment-5767781260
Maybe I'm misreading this conversation but it seems the original as was to support Postgres style explicit `AS MATERIALIZED` directives, but then we went into discussing automatically doing this for CTEs based on heuristics/optimizers to e.g. speed up TPC-DS. The controversial part of this is deciding when to materialize vs. inline w/o many stats to guide us. But I don't see *any* reason we wouldn't implement support for explicit `AS MATERIALIZED` directives? That would: 1. Unlock use cases where users are able to include the directive explicitly. 2. Allow us to write the operators / implementation to actually materialize the CTEs (there's probably discussion to be had about spilling and other implementation details). Then we can spit off the work of writing an optimizer / heuristic / flag for automatically materializing CTEs even when the user didn't explicitly request it. @neilconway @zhuqi-lucas @nathanb9 wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
