sanchay0 commented on issue #11894: URL: https://github.com/apache/iceberg/issues/11894#issuecomment-2628445022
Leaving an update here for others who might run into this in the future as this was rather tricky to debug. We are currently using Iceberg 0.11.x on Flink 1.11, there is a bug that was fixed in https://github.com/apache/iceberg/pull/2745/ to explicitly set operator UID in the Iceberg sink. If you don't explicitly set the UID and your job graph changes while deploying a change with a checkpoint, Flink won't be able to restore the previous Flink sink operator state, specifically the committer operator state. However, you can circumvent this issue by using the `--allowNonRestoredState` flag. During the restore process, Flink uses the sink state to verify whether checkpointed files were actually committed. Using `--allowNonRestoredState` can lead to data loss because the Iceberg commit might have failed in the last completed checkpoint. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
