Xuanwo commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-1911446953
Hi @tustvold, thank you for initiating this discussion! I will do my best to offer a multifaceted response with different hat. ## Put iceberg-rust developers hat on As @liurenjie1024 mentioned, iceberg-rust features its own `FileIO` interface to abstract IO operations. `OpenDAL` and `object_store` are merely implementation details with no current plans for external exposure. It's fine to integrate with `object_store`, as that is precisely what we created `FileIO` for. However, it's important to note that we are in the initial stages of this project: currently focusing on the first release and implementing read/write capabilities. Here are some remarks regarding the object_store feature set: > A flexible [configuration system](https://docs.rs/object_store/latest/object_store/#configuration-system) developed in partnership with, and used by both the polars and delta-rs communities iceberg-rust is aligned with Iceberg and PyIceberg, sharing the same configuration logic; therefore, the object_store's configuration system is redundant for our purposes. > Support for [conditional writes](https://docs.rs/object_store/latest/object_store/#conditional-put), which would allow iceberg-rs to support multiple concurrent writers directly against object storage, without needing an external catalog While the conditional put feature offers certain advantages, it may not be as crucial for our current use cases in iceberg-rust, where integration with a catalog like Hive or REST is more common. As an iceberg-rust developer, I am eager to unlock more potential within the project. ## Put OpenDAL maintianer hat on Firstly, [opendal](https://github.com/apache/opendal) and [object_store](https://crates.io/crates/object_store) are not competitors. (And remember, I'm also a contributor to `object_store`!) Rather than discussing replacements, I'd prefer to explore how we can coexist to offer our users more choices and possibilities. I believe `opendal` integrates seamlessly with `object_store`, which is why our community created [`object_store_opendal`](https://github.com/apache/opendal/tree/main/integrations/object_store), enabling users to utilize `opendal` as an implementation of `object_store`. Here are a few reasons why OpenDAL is beneficial for iceberg-rust. - OpenDAL offers native support for OSS, B2, HDFS, and WebHDFS in addition to the existing S3, GCS, and AzBlob. All have passed identical [behavior test suites](https://github.com/apache/opendal/actions/runs/7662788480), simplifying integration for users without fear of unexpected breaking. - OpenDAL offers a comprehensive API that supports range retrieval and conditional fetching through the robust [`read_with()`](https://docs.rs/opendal/latest/opendal/operator_futures/struct.FutureRead.html) function. - OpenDAL enables users to freely utilize its API. For instance, they can directly use [`Writer`](https://docs.rs/opendal/latest/opendal/struct.Writer.html) without needing to understand MultipartUpload. - OpenDAL features powerful layers such as [retry](https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html), [logging](https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html), [tracing](https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html), [metrics](https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html), [prometheus](https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusLayer.html), [timeout](https://docs.rs/opendal/latest/opendal/layers/struct.TimeoutLayer.html), and [more](https://docs.rs/opendal/latest/opendal/layers/index.html) that can significantly reduce the workload typically associated with managing these aspects manually. - OpenDAL features [`object_store_opendal`](https://github.com/apache/opendal/tree/main/integrations/object_store) integration, enabling seamless connection to existing `object_store`-based systems. I also found some places that OpenDAL can improve (Thanks @tustvold!): - https://github.com/apache/opendal/issues/4074 - https://github.com/apache/opendal/issues/4075 As an OpenDAL maintainer, I believe OpenDAL offers features that could be beneficial for iceberg-rust, potentially simplifying some aspects of storage management. And I will be happy to collaborate with `object_store` to ensure the success of iceberg-rust. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
