alamb commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-1911845190
Thank you all -- this is a great conversation. > I entirely agree, I guess I was more suggesting that the IO abstraction mirror object_store as this is what both the upstream crates use and expect, and what the underlying stores provide. If people then wanted additional backend support they could plug OpenDAL into this interface? I took a look at the [FileIO](https://github.com/apache/iceberg-rust/blob/3b5c35ebc0b6e47bfaf74167711e7b605d994ab3/crates/iceberg/src/io.rs#L146) interface that @liurenjie1024 and @Xuanwo pointed it. Eventually they seem to provide something that implements `AsyncRead` and `AsyncWrite` While it is true that `AsyncRead` and `AsyncWrite`'s interfaces (seek, random IO, etc) can be used in such a way that would perform very poorly for remote object storage, I think if users are judicious and provide sufficients hints, and buffer the reads the performance difference will be negligible. The "benefit" that one might get from using `object_store` is that its API is more opinionated and makes it very awkward to use poorly In my opinon, the use of OpenDAL to connect to more storage systems other than object stores is pretty compelling. Perhaps as you proceed integrating iceberg-rust with arrow-rs/parquet/datafusion we will learn more about how these various systems can be integrated and if any adjustments need to be made, either in OpenDAL or object_store or downstream in some other crates -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
