vladislav-sidorovich commented on PR #15407: URL: https://github.com/apache/iceberg/pull/15407#issuecomment-6167097315
Thanks @kevinjqliu and @anoopj for the thoughtful feedback and for being open to getting this PR merged! Test files. Let's discuss it in the thread with your comment to align the state we need to get this PR ready to merge. Regarding the broader question of whether `iceberg-delta-lake` belongs in the Iceberg repository and the concrete use cases driving it, I’d love to share why we are investing in this module and why it fills a critical gap that UniForm and XTable don't cover: ### 1. Pull-Based Migration (Iceberg-native) vs. Push-Based Interoperability (UniForm) @kevinjqliu raised a great point that this module spends a lot of effort understanding Delta protocol/Kernel APIs. However, from a user and operational standpoint, **where the tool lives dictates how a migration is executed**: * **Why users can't always use UniForm:** UniForm is a *push-based* interoperability feature on the Delta side—it requires write access to alter the source Delta table properties, upgrade the table protocol, and trigger metadata generation from a Delta engine. In many real-world enterprise migrations we see, the data engineering team moving to Iceberg either **cannot modify legacy production Delta tables** (due to strict governance or freeze policies) or only has read access to the underlying object storage. * **Why XTable isn't a drop-in replacement:** Apache XTable (Incubating) is an ambitious omni-directional sync tool, which brings additional operational overhead and is still in incubation. (as of today, it has gaps that block real-world usage in migration scenarios) * **The role of `iceberg-delta-lake`:** Just as Iceberg provides native migration utilities for Hive and raw Parquet tables (`migrate`, `snapshot`, `add_files`), having a **pull-based, zero-copy migration module natively in Iceberg** allows users to convert Delta tables into native Iceberg v2/v3 tables (including translating Delta DVs to Iceberg Puffin DVs and preserving snapshot history) purely from the Iceberg side without touching the source Delta environment. ### 2. Bridging the Gap to Iceberg v4 @anoopj's point about Iceberg v4 and Delta eventually sharing a unified adaptive metadata tree on disk is super exciting and will be a huge win for the ecosystem long-term! At the same time, we have strong, immediate demand from customers today who are migrating existing Delta tables (Reader v1–v3 / Writer v1–v7) into Iceberg v2 and v3. This module serves as the practical migration path for those production workloads today. If v4 eventually makes translation obsolete down the road, sunsetting it at that point will be a natural evolution. ### 3. Real-World Migration Complexity & Planned Spark Procedure Migrating production tables is rarely a single-step metadata swap—it involves catalog registration, path handling, and incremental catch-up during dual-write windows (I recently covered some of these real-world nuances in an article on [migrating tables to GCS and BigLake Iceberg REST Catalog](https://medium.com/google-cloud/migrating-apache-iceberg-tables-to-gcs-and-biglake-iceberg-rest-catalog-c68399d7a03d)). For Delta -> Iceberg, customers currently lack a convenient, native tool. Once this foundation is merged, my immediate goals are: 1. **Add a Spark Procedure** (similar to [`add_files`](https://iceberg.apache.org/docs/latest/spark-procedures/#add_files) / `snapshot` for Hive migrations) so users can run Delta-to-Iceberg migrations directly via Spark SQL. An example I have in mind looks like this: (of course it's the next step and we will discuss it separately) ``` CALL spark_catalog.system.confert_to_iceberg( table_type => 'delta', source_location => 'path/to/table', ... ); ``` 2. **Publish a comprehensive How-To Guide** walking the community through end-to-end Delta-to-Iceberg migration scenarios. ### 4. Commitment to Ownership & Maintenance To address the concern about the module going inactive like it did in 2023: **I am committed to actively maintaining and supporting this module.** We have a structured backlog tracked in [#17146](https://github.com/apache/iceberg/issues/17146) and our [Development Plan Doc](https://docs.google.com/document/d/1eWH7N_9Mo2b1cDm1jFM9R7I16iYa5xZUvtU_9eLozjA/edit?tab=t.0#heading=h.oc0qb4rn6qup) (covering removing internal Delta Kernel usages, incremental sync, column mapping, and schema evolution). I'm also happy to chime in on the `dev@` mailing list thread so we have this context recorded for the wider community. I'll push the commit removing the extra test files shortly! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
