tongwai-wong-appier commented on issue #13763: URL: https://github.com/apache/iceberg/issues/13763#issuecomment-4922110838
Following up on my earlier comment and #16282 — this still reproduces, now on **Iceberg 1.11.0** (previously 1.10.1), JDBC catalog + GCS, append-only table. Same symptom (one physical parquet registered in two adjacent snapshots), and this incident gave us more detail on the trigger. **What we observed** Two adjacent snapshots (~68s apart), committed by the **same** coordinator task, with an **identical** `kafka.connect.valid-through-ts`: ``` snapshot A commit-id 25210aa1… +47 data files snapshot B commit-id deb2cf2b… +185 data files → 45 file_paths are byte-identical across A and B (~4.6M rows double-counted) ``` Decoding the control topic for that window: - the 45 duplicated files appear as `DATA_WRITTEN` **exactly once**, under commit-id **A**; - they do **not** appear under commit-id **B** at all. So this looks like **coordinator-side re-registration**, not a worker re-sending the file. We also confirmed the coordinator task never lost leadership and its `iceberg-coord` thread never restarted (continuous consumer client-id). What we did see at that moment was a plain **`-coord` consumer-group rebalance mid commit-cycle** (control-topic partition revoke → re-join → re-add; the group had a second, 0-partition member at the time), followed by ~90 `received … when no commit in progress` log lines for commit-id **A**. **Our reading of how the replay becomes a duplicate (hypothesis, not fully proven)** We haven't nailed the exact millisecond ordering, but from the code the following seems consistent with what we saw: - On re-join, the control consumer resumes from the **last committed offset**; since the in-flight commit hadn't committed yet, that boundary is the *previous* cycle, so the round's events get **replayed**. - `CommitState.isCommitReady()` sums `assignments().size()` over `readyBuffer` with no dedup / no distinct-partition tracking, so replayed `DATA_COMPLETE` may inflate the count and let the commit fire early (snapshot A) while the round is still being re-delivered. - The re-delivered tail (`DATA_WRITTEN`) then arrives after that commit ended (`currentCommitId == null`); `addResponse()` buffers it unconditionally and nothing resets `commitBuffer` on the rebalance (the control consumer `subscribe()` has no `ConsumerRebalanceListener`), so those events get swept into the next commit (snapshot B). If that reading is right, then on top of the non-atomic offset advancement from #16282, there also seems to be an in-cluster asymmetry: a rebalance rewinds the consumer position but does not reset the coordinator's in-memory `CommitState`. Would appreciate any thoughts on whether this matches your understanding, or points to a different root cause. Happy to share the decoded control-topic events, snapshot manifests, and coordinator logs. Refs (Iceberg 1.11.0, `org.apache.iceberg.connect.channel`): `CommitState.isCommitReady` / `addReady` / `addResponse`; `Channel.commitConsumerOffsets` + `subscribe`; `Coordinator.commitToTable`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
