Xuanwo commented on issue #1314: URL: https://github.com/apache/iceberg-rust/issues/1314#issuecomment-2919079557
Sorry for I didn't join the party earlier. --- I agree with the idea of making `Storage` a trait and turning `FileIO` into a wrapper that can be used throughout the entire project. The `Storage` trait can be dedicated solely to the `iceberg` use case, while also enabling users to implement their own custom IO solutions. I’m aware that some vendors prefer to use their own high-performance IO implementations, optimized with io-uring and RDMA, which are not currently supported by either `object_store` or `opendal`. Introducing a dedicated storage trait would make this possible. This trait also prevents iceberg-rust from depending directly on any implementations, allowing it to evolve according to our own needs without being constrained by upstream dependencies. In addition to `Storage` itself, we will also need something like a `StorageBuilder` to accept storage options passed from the REST Catalog or Pyiceberg. It would also be useful to have a `StorageRegistry` that holds different instances of `s3://bucket/path/to/file` to avoid rebuilding them every time. Like what we do for `Catalog`, maybe we can add `Storage` trait and `FileIO` struct in the core and have something like `iceberg-storage-opendal` and `iceberg-storage-object-store`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
