gabotechs opened a new issue, #3029: URL: https://github.com/apache/iceberg-rust/issues/3029
Continuing the https://github.com/apache/iceberg-rust/discussions/2992 discussion in an issue as requested by @andygrove, so that updates get sent to the mailing list. Citing the original discussion description: --- I’d like to explore decoupling the development lifecycle of iceberg-datafusion from iceberg-rust by moving the existing [DataFusion integration](https://github.com/apache/iceberg-rust/tree/main/crates/integrations/datafusion) into a crate maintained in a separate repository. As discussed in https://github.com/datafusion-contrib/datafusion-distributed/issues/599, we are exploring incorporating the integration into [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed), making native distributed Iceberg support a core offering of that project. Another option worth exploring would be to create a dedicated repository, such as datafusion-contrib/datafusion-iceberg. Regardless of its eventual home, the goal would be to allow the community to advance the Iceberg+DataFusion integration without adding review pressure for iceberg-rust maintainers. The integration could evolve at its own pace and be maintained by a different or broader group of contributors. Some reasons why this may be worth considering: There appears to be a vision of making iceberg-rust engine-agnostic (https://github.com/apache/iceberg-rust/pull/2613#issuecomment-4830211788, cc @blackmwk). Several integration-related PRs have had difficulty receiving review attention in this repository, such as https://github.com/apache/iceberg-rust/pull/2671 and https://github.com/apache/iceberg-rust/pull/2521. Other DataFusion-related projects have expressed interest in hosting an Iceberg integration, including https://github.com/apache/datafusion-ballista/issues/2174 and https://github.com/datafusion-contrib/datafusion-distributed/issues/599. I’d like to hear from the community (particularly the iceberg-rust maintainers) about the possibility of moving iceberg-datafusion out of the iceberg-rust repository. This could allow projects such as [Ballista](https://github.com/apache/datafusion-ballista) and [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed) to share common functionality and contribute to the integration independently of the iceberg-rust release and review lifecycle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
