linhr commented on issue #172:
URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2869546190
Having object_store as an OpenDAL layer seems a good idea!
Besides supporting object_store, it seems `FileIO` currently only supports a
few specific OpenDAL layers listed as `Storage` enum variants. I'm wondering if
the storage can be generalized to support **any** OpenDAL layers.
It seems to me that `FileIO` and the underlying `Storage` right now is
managing the configuration of storage layers. (There is a slight inconsistency
though, since `Storage::Memory(Operator)` manages the OpenDAL operator while
the other variants manage configuration such as `S3Config`.) I'm wondering if
we can consider an alternative of the design where `Storage` manages OpenDAL
operator instances with an optional builder that constructs `Storage` from
configuration. This is the idea of "dependency injection" so that the library
can be decoupled from, and more easily integrated with, other systems.
Specifically, I notice the following code for `FileIO`.
```rust
impl FileIO {
/// Convert FileIO into [`FileIOBuilder`] which used to build this
FileIO.
///
/// This function is useful when you want serialize and deserialize
FileIO across
/// distributed systems.
pub fn into_builder(self) -> FileIOBuilder {
self.builder
}
}
```
I'm not sure if we'd like to ensure that `FileIO` is serializable. The end
application (e.g. Sail) would likely have their own configuration systems and
their own way for distributed coordination. The only thing they need is a way
to construct `FileIO` in memory and inject their own storage implementations.
This makes the Iceberg library more flexible with a separation of concerns. It
also unblocks use cases such as automatic credential rotation, where credential
management is a implementation detail inside `dyn ObjectStore` without passing
in something like `S3Config` to `Storage` or `FileIO`.
Maybe this topic is more suitable as a separate GitHub issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
