belldoor commented on issue #8916: URL: https://github.com/apache/iceberg/issues/8916#issuecomment-2475756495
I have identified the root cause of the problem. The Iceberg connector commits to the Iceberg table at the point when a checkpoint is successfully completed. In my code, since I did not enable the checkpoint, the write commit did not occur. If you are experiencing a similar issue, please check whether the checkpoint is enabled. I got a hint for this idea from the metrics section of the [Iceberg Flink documentation](https://iceberg.apache.org/docs/1.7.0/flink-writes/#metrics). > Iceberg commit happened after successful Flink checkpoint in the notifyCheckpointComplete callback. It could happen that Iceberg commits failed (for whatever reason), while Flink checkpoints succeeding. So all you need to do is enable checkpointing like in the code below. ```py env.enable_checkpointing(60000) # 60s checkpoint_config = env.get_checkpoint_config() checkpoint_config.set_checkpoint_storage_dir('file:///tmp/checkpoint') ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
