kumarpritam863 commented on issue #13763:
URL: https://github.com/apache/iceberg/issues/13763#issuecomment-4436811811
Hi @tongwai-wong-appier the de-duplication is not only controlled by the
control topic offsets commit by the coordinator but also the coordinator puts
the control topic offset in the table.
So there are three guarantees that are providing exactly once:
1. Workers commit the file metadata and offset in the same transaction and
coordinator uses read_committed to read from the control topic.
2. Iceberg guarantees that now two writers write the same snapshot, even if
both of them see the sanpshotId = 100 and try to make it 101, one is bound to
fail and retry.
3. Coordinator when commits the data to iceberg, it also puts the control
topic offsets as a metadata in the snapshot.
So even if during coordinator switchover after a Kafka timeout: the old
coordinator can successfully commit file X, but before the control-topic offset
is advanced, if new coordinator re-consume stale DATA_WRITTEN(file X) and try
to commit there will be two scenarios:
1. Layer 1 — Offset Validation (Kafka Connect level):
The coordinator attaches a SnapshotAncestryValidator to every
append/row-delta operation that checks committed offsets stored in the snapshot
summary. If Coordinator A commits first, Coordinator B's validator sees the
offsets already advanced and raises a
ValidationException. This is not retried — it's a logical conflict, not a
transient failure.
2.Layer 2 — Metadata Version Check (Iceberg Core level):
BaseMetastoreTableOperations.commit() compares the base metadata pointer
against the current one. If another commit changed the table metadata, it
throws CommitFailedException. This is retried up to 4 times with exponential
backoff (100ms → 5s total).
There were some issues with this offset validation which were fixed in these
two PR's. Make sure you are using a version which has these changes
https://github.com/apache/iceberg/pull/14525 and
https://github.com/apache/iceberg/pull/14510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
