xudong963 commented on issue #25775: URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5862192889
Streaming file discovery is closer to what I had in mind. In Atlas(our internal database built on DF), we have an internal `FileScanExec` that pulls files a page at a time during execution, builds a `FileScanConfig` for each page, and delegates the actual reads to DataFusion’s file scan. Our candidate `FileSet` is still selected from an index during planning, so it does not yet cover lazy Delta log replay, but the execution pattern seems similar. I would recommend a custom execution plan in delta-rs first: it could consume the metadata stream, apply the runtime filter, and create file scans for the surviving files. That would keep Delta snapshot and log semantics in delta-rs. If the implementation reveals a specific missing DataFusion API, we could then discuss that narrower extension upstream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
