Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2309781356 > @zhongqishang: Could you please rebase? @pvary Already rebased, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1728933367 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,45 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2307049114 @zhongqishang: Could you please rebase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1719379441 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,45 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1719379441 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,45 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1719368722 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,45 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2290970014 @zhongqishang: Thanks for the patience. I revisited the PR again, and found one potential issue. Could you please check it? I think we are getting there. Thanks @gyfora for the clarific

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1718186250 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,45 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1718184548 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,45 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

gyfora commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1715155476 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1695119335 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1686446191 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1685640900 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2238165277 > @zhongqishang @pvary I have a uber question. > > let's say checkpoint N was cancelled or timed out and checkpoint N+1 completed successfully. In this case, do we know all the writ

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683818497 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683819074 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergFilesCommitter.java: ## @@ -204,7 +207,7 @@ public void testCommitTxn() throws Except

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683742893 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2237743264 @zhongqishang @pvary I have a uber question. let's say checkpoint N was cancelled or timed out and checkpoint N+1 completed successfully. In this case, do we know all the writer

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683576927 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergFilesCommitter.java: ## @@ -204,7 +207,7 @@ public void testCommitTxn() throws Exception

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683550064 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683536734 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683536734 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683518028 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683534328 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683519816 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

stevenzwu commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1683518028 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Overr

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on PR #10526: URL: https://github.com/apache/iceberg/pull/10526#issuecomment-2236539203 @zhongqishang: Finally found time to start the upstream discussion: https://lists.apache.org/thread/n5c85hd7psf2tmgych6scmynonscp2q4 -- This is an automated message from the Apache Git

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650864804 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650851730 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650849668 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650391006 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Ov

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650379369 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -426,30 +425,44 @@ private void commitOperation( } @Override

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1650253742 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -259,28 +268,28 @@ private void commitUpToCheckpoint( l

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1648478721 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -259,28 +268,28 @@ private void commitUpToCheckpoint( l

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

zhongqishang commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1648452582 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergFilesCommitter.java: ## @@ -887,6 +921,55 @@ public void testCommitTwoCheckpointsInSi

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1648420475 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergFilesCommitter.java: ## @@ -887,6 +921,55 @@ public void testCommitTwoCheckpointsInSingleTxn

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1647508483 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/SimpleDataUtil.java: ## @@ -82,6 +83,13 @@ private SimpleDataUtil() {} Types.NestedField.optional

Re: [PR] Flink: Fix duplicate data in Flink's upsert writer for format V2 [iceberg]

pvary commented on code in PR #10526: URL: https://github.com/apache/iceberg/pull/10526#discussion_r1647501589 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java: ## @@ -448,8 +452,8 @@ private byte[] writeToManifest(long checkpointId) th