alamb commented on issue #25775: URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5857187783
@mohitgurav20 I am not sure what exactly you are proposing @ion-elgreco I feel like the suggestion of child nodes might be confusing. From your description, I can't tell if you 1. want is a way to add a check before the next file is opened dynamically and potentially skip it based on information you learn during runtime (like a DynamicFilter) 2. Want a way to add new files to the scan at runtime that was not known at planning time 3. Something else If it is adding a dynamic check (option 1), there is already a bunch of examples of dynamically pruning down a list of files known at planning time with information discovered during runtime (specifically in the parquet opener / file source --- see https://github.com/apache/datafusion/blob/f75c5032af7693cc92583502e57f2afddf140325/datafusion/pruning/src/file_pruner.rs#L37-L36 and its uses for example) If it is something else, we can find something else I think @adriangb @zhuqi-lucas @haohuaijin and @xudong963 have worked in this area extensively -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
