BlakeOrth commented on issue #172: URL: https://github.com/apache/iceberg-rust/issues/172#issuecomment-2471199556
Hi, I'm a relatively new user of the `icberg-rust` crate(s) and was hoping I could bring this discussion back to get some movement here. While Iceberg is a relatively new ecosystem for me, I'm reasonably well versed in both object stores as well as parquet. As such, I was hoping to be able to use the `object_store` crate that @tustvold surfaced here with `iceberg-rust` so I could have a unified storage backend for an application I'm working on. Unfortunately, since `iceberg-rust` relies on the `FileIO` struct for all IO operations I don't immediately see a way to use anything except the provided implementations, outside of reimplementing the existing parquet writer and all of its column statistic functionality. I've read over the discussion here, as well as #356, reviewed the ethos behind `FileIO` from @Fokko's link, and have been looking into the various uses of `FileIO` throughout the code. While the easy and least disruptive answer here seems to be simply exposing `FileIO` as a trait instead of a struct, I question whether or not that's the best path forward. I understand the desire to retain `FileIO` for uniformity across the Iceberg ecosystem, however there also seems to be value in adopting the `object_store` trait for uniformity across the rust Arrow/Parquet ecosystem. Given the very small API surface area of `FileIO` (`newInput()`, `newOutput()`, `delete()`) I'm wondering how everyone would feel about adopting `object_store` as the core storage solution for `iceberg-rust` and implementing `FileIO` on top of `object_store` instead of OpenDAL directly. It seems like this would allow `iceberg-rust` to take advantage of the robust and well tested ecosystem surrounding `object_store` , including all of its existing integrations like OpenDAL, while retaining the option for Iceberg users to leverage the familiar `FileIO` if they wanted. @liurenjie1024, do you think this would satisfy your concerns you voiced in #356? I'm not sure if @tustvold still has the time to open a PR, but I would be happy to help contribute towards this effort as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
