blackmwk commented on PR #2613: URL: https://github.com/apache/iceberg-rust/pull/2613#issuecomment-4830211788
> > Thanks @NoahKusaba for this pr, but I don't think it's the right direction to maintain such a huge integration in this repo. Currently the review resources is quite limited, and most committers are not familiar with ballista. I think it would be better to put it in ballista repo > > @blackmwk This still seems like the logical place to put it. This integration directly builds on the datafusion-iceberg integration that lives in this repo to interact with Iceberg. There already seems precedence for that with the Playground integration that lives in /integrations, also seems like other integrations with iceberg are parked here such as cache-moka. > > At the very least I'll look to split up the iceberg-datafusion extensions in separate PR's over the coming weeks. (still tightening up the full implementation). The position of iceberg rust is to provide a library to be integrated into all differeent compute engines. We should not put the integration of other engines due into this repo for several reasons: 1. The resource of maintaners is quite limited, so we should focus the core library part. 2. Most of the maintainers are not familiar with ballista. 3. Rust, unlike java, which provided great runtime reflections to load libraries dynamically, currently has no mature solution for that. We put the integration of datafusion in this repo initially to guide api design and provide a reference implementation for other engines. I think we should discuss moving the datafusion integration into other repos(like datafusion repo) when the implementation is mature enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
