Beutlin commented on issue #18098: URL: https://github.com/apache/iceberg/issues/18098#issuecomment-5671414528
Thanks for the quick response! To clarify the scenario — there was no deliberate restore/rollback of anything on our side. It's not that the source was restored to an earlier state while the table stayed current; rather, this was a fully stateless restart: no checkpoint or savepoint was restored at all (isRestored() == false). The Flink job simply started fresh, so its checkpoint counter began again at a low number — which, from the table's perspective, looks like "the source went back in time," but nothing was actually rolled back on either side. The table itself was never touched or reset; it kept its full, untouched commit history from the previous run. So restoring the table to match isn't really applicable here — there's no "original state" to restore to, since the previous run's commits were all correct and intended; we just don't want them to block new, unrelated commits from a job that has no relationship to that old checkpoint numbering. Re: updating uidSuffix — that matches our understanding too, and sounds like the right-shaped workaround (rotate the operator UID on every stateless deployment so the (job-id, operator-id) lookup in SinkUtil.getMaxCommittedCheckpointId no longer matches old snapshots). We ended up reverting to the legacy FlinkSink API instead (which guards that lookup with isRestored()), but rotating uidSuffix seems like a cleaner fix if we want to stay on IcebergSink. Given that, would it make sense for IcebergSink's commit path to also gate the getMaxCommittedCheckpointId lookup on isRestored(), the way the legacy committer does? That would make a stateless restart behave consistently regardless of whether uidSuffix happens to be reused. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
