liurenjie1024 commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-1911328016
Hi, @tustvold @alamb Thanks for this proposal and write up, [object_store](https://crates.io/crates/object_store) looks great to me! In iceberg's design, all file ios are hidden under the [`FileIO`](https://github.com/apache/iceberg-rust/blob/3b5c35ebc0b6e47bfaf74167711e7b605d994ab3/crates/iceberg/src/io.rs#L146) interface, and the backends, i.e. `OpenDAL` or `object_store` are not directly exposed to user, so I think we can integrate it without any breaking changes. Currently `OpenDAL` works well for us and we are focusing on implementing more features for `iceberg-rust`, so it may take a while for us to evaluate `object_store` and integrate it into this crate. > First-party integration with [arrow-rs](https://docs.rs/arrow-csv/latest/arrow_csv/reader/index.html#async-usage), [parquet](https://docs.rs/parquet/50.0.0/parquet/arrow/async_reader/struct.ParquetObjectReader.html), [DataFusion](https://docs.rs/datafusion/latest/datafusion/datasource/object_store/trait.ObjectStoreRegistry.html) and [polars](https://docs.rs/polars-io/0.36.2/polars_io/cloud/fn.build_object_store.html), including sophisticated [vectored](https://docs.rs/object_store/latest/object_store/#vectored-read) and [streaming](https://docs.rs/object_store/latest/object_store/struct.GetResult.html#method.into_stream) IO I'm quite interested in this since we are about to add support for file reader/writer, which will heavily depend on `arrow-rs`, `parquet`, etc, so I think `object_store` is quite promising. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
