kevinjqliu commented on PR #15407: URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6152231331
Thanks @vladislav-sidorovich for driving this, and to everyone who's put in multiple rounds of review here. I want to zoom out a bit on the direction of `iceberg-delta-lake` and whether it makes sense for it to live in the Iceberg project. Most of what this module does is understand Delta: its log, protocol features, and DV format. The Iceberg side is just writing a table. Keeping it correct means tracking Delta's protocol and Delta Kernel's APIs, which seems like a better fit for a project that already owns that knowledge. We've touched on this a couple of times on the devlist: - In ["Inactive modules?"](https://lists.apache.org/thread/fv0hfyjo807yp1vv38qmf7qgprnssnmd), the module came up as a deprecation candidate since there hadn't been meaningful activity since 2023. It was kept based on plans to upgrade it, not a broader decision about whether it belongs here. - In [the devlist thread for this PR](https://lists.apache.org/thread/z48hr4m28xh6k0ptzrhl52kyxc9r3vrq), the question of sunsetting it came up again, since Delta UniForm can already generate Iceberg metadata (incrementally, with column mapping support) and IcebergCompatV3 should lift the DV limitation. That thread didn't get much wider input. So i don't think we've really settled this as a community. There are other natural homes for this: Delta itself (UniForm, or contributing this converter there, next to Delta Kernel) or a cross-format project like XTable. Given that, i think it's worth reevaluating whether there's still a benefit to having this module in Iceberg at all. If we do keep it here, do we have concrete use cases from the community that need it? Would be great to hear from folks who are using this today or planning to. To be clear, i'm not trying to block this PR, and we can keep iterating on it here. I just think we should be deliberate as a community about whether this belongs in Iceberg before we build more on top of it. Happy to help facilitate the discussion, i can either bump the devlist threads above or start a new thread so we can get more input. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
