sanchay0 commented on issue #11894: URL: https://github.com/apache/iceberg/issues/11894#issuecomment-2568342110
@pvary I did restart from a savepoint. I mentioned that in the description > I restart the job from the latest savepoint, which is committed at the Kafka source The problem is that Iceberg file committer did not finish writing data to my object store (S3) for the older Kafka offset (say `x`) that was committed to the savepoint. Now say offset `x + 1` is written to the next savepoint before Iceberg file committer finishes writing the data for the previous offset `x`, upon restart the data that was read from `x` offset will never be written out to the Iceberg table and hence lost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
