[I] Handling Updates on Partition Columns in Iceberg with Flink CDC [iceberg]

[via GitHub]

a8356555 opened a new issue, #11573:
URL: https://github.com/apache/iceberg/issues/11573

   ### Apache Iceberg version
   
   1.5.2
   
   ### Query engine
   
   Athena
   
   ### Please describe the bug 🐞
   
   Hi,
   
   I'm using MySQL Flink CDC with Iceberg 1.5.2 and Flink 1.16. I have a table 
partitioned by the status column, but this column is subject to updates. When 
an update occurs, I encounter duplicate records in the Iceberg table, which is 
not the desired behavior.
   
   Is there a way to properly handle updates on a partition column in Iceberg 
to avoid duplicates?
   
   here is the sql of my flink CDC
   ```
   CREATE CATALOG glue_catalog WITH (
       'type'='iceberg', 
       'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
       'warehouse'='s3://my-bucket'
   );
   
   CREATE TABLE mysql_cdc_source
   (
       id  INT,
       status INT,
       value  INT,
       ...,
       PRIMARY KEY (id) NOT ENFORCED
   ) WITH (
       'connector' = 'mysql-cdc',
       'hostname' = 'mysql',
       'port' = '3306',
       'username' = 'XXX',
       'password' = 'XXX',
       'database-name' = 'XXX',
       'table-name' = 'XXX',
       'debezium.snapshot.mode' = 'when_needed',
       'server-id' = '1'
   );
   
   CREATE TABLE IF NOT EXISTS glue_catalog.my_db.my_table(
       `id` INT NOT NULL,
       `status` INT,
       `value`  INT,
       ...,
   iceberg)
   PRIMARY KEY (id,status) NOT ENFORCED
   ) PARTITIONED BY (
       status
   ) WITH (
       'format-version'='2',
       'write.metadata.delete-after-commit.enabled'='true',
       'write.upsert.enabled'='true',
       'write.delete.mode'='merge-on-read',
       'write.merge.mode'='merge-on-read',
       'write.update.mode'='merge-on-read'
   );
   
   INSERT INTO glue_catalog.my_db.my_table
   SELECT
       *
   FROM mysql_cdc_source;
   ```
   
   data before updating mysql: 
   
![image](https://github.com/user-attachments/assets/9ae9781a-d1bd-4719-9f43-35604314325a)
   
   data after updating mysql (duplicated row showed up): 
   ![截圖 2024-11-18 中午12 19 
24](https://github.com/user-attachments/assets/1ed1410f-6329-425d-8264-d523c6d90f63)
   
   get same results using spark 3.4:
   <img width="144" alt="image" 
src="https://github.com/user-attachments/assets/78b28982-ebf0-4a53-9bf6-248060b847af";>
   
   query `glue_catalog.my_db.my_table.files` using spark 3.4
   
![image](https://github.com/user-attachments/assets/6b4c904a-6501-43ca-8cb7-d479d7b949ca)
   
   
   Thanks!
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org