liurenjie1024 commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2496515292
> I _think_ that should work, the DataFusion wrapper can just hook the iceberg metadata operations into via that StorageProvider trait, and then use the DataFusion machinery directly for the actual file I/O. I presume this would be possible, or does this library expect to also perform the file I/O? I think that depends. This library provides several functionality for reading: 1. Planning data files to read for table scan. This involves access to metadata files, and it use `FileIO` to do that. 2. A builtin reader to convert planned table scan tasks to arrow record batch stream. In this case, we also use `FileIO` to access data files. For integration with query engines like datafusion, datafusion only need to use planning file api, and perform scanning by itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
