ion-elgreco commented on issue #25775: URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5860159091
> [@ion-elgreco](https://github.com/ion-elgreco) I feel like the suggestion of child nodes might be confusing. From your description, I can't tell if you > > 1. want is a way to add a check before the next file is opened dynamically and potentially skip it based on information you learn during runtime (like a DynamicFilter) > 2. Want a way to add new files to the scan at runtime that was not known at planning time > 3. Something else > > If it is adding a dynamic check (option 1), there is already a bunch of examples of dynamically pruning down a list of files known at planning time with information discovered during runtime (specifically in the parquet opener / file source --- see https://github.com/apache/datafusion/blob/f75c5032af7693cc92583502e57f2afddf140325/datafusion/pruning/src/file_pruner.rs#L37-L36 and its uses for example) > > If it is something else, we can find something else > > I think [@adriangb](https://github.com/adriangb) [@zhuqi-lucas](https://github.com/zhuqi-lucas) [@haohuaijin](https://github.com/haohuaijin) and [@xudong963](https://github.com/xudong963) have worked in this area extensively Option 1 is actually what I had refactored it to yesterday, with DynamicFilterPhysicalExpr to prune the parquetSource. The idea I wanted to explore is, instead of having a **static** list of files, to have a streamed list of files that can be coming from another plan node. For example, our log replay could be a lazy execution plan that streams the file paths to the parquet node. I think option 2 reflects that the best -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
