allthingssecurity opened a new pull request, #18803:
URL: https://github.com/apache/iceberg/pull/18803
When the sink commits to a branch (`iceberg.tables.default-commit-branch` or
a table's `commit-branch`) and that branch does not exist,
`Coordinator.lastCommittedOffsetsForTable` reads the offsets from
`table.snapshot(branch)`, which is `null`, so it expects `{}`. The commit
itself resolves the parent with `SnapshotUtil.latestSnapshot(TableMetadata,
String)`, which falls back to the current snapshot when the branch ref is
missing, and the offset validator then finds the connector's offsets in that
snapshot's ancestry. The two disagree and every commit fails with
`ValidationException: ... stale offsets: Expected: {} Committed: {0=n}`; the
coordinator stops, the task fails again after each restart, and nothing is
committed until the branch is re-created by hand. This happens in the
audit-branch (WAP) workflow from the branching docs when the branch is dropped
after it was published to main, when a branch with `max-ref-age` expires, or
when a connector that wrote to main is switched
to a new branch. This reads the offsets from the same snapshot the commit
uses, so the first commit re-creates the branch from main and continues main's
offsets; the behaviour for an existing branch or for main is unchanged.
Test plan: new
`TestCoordinator#testCommitToMissingBranchUsesOffsetsFromCurrentSnapshot`.
Without the change it fails with `ValidationException: Snapshot ancestry
validation failed: Cannot commit to db.tbl, stale offsets: Expected: {}
Committed: {0=1}`; with it, it passes and the branch is created from main's
head with offsets `{"0":3}`. End to end, a harness driving the released 1.12.0
`Coordinator`, `Worker` and `SinkWriter` against a real table (publish the
branch to main, drop it, keep writing) failed every one of 3 commit cycles in 3
of 3 runs while the source offsets advanced; keeping the branch committed all
records. With the same one-line change applied to the released 1.12.0
`Coordinator`, the dropped-branch run committed all 12 records and re-created
the branch, with no duplicates. `spotlessCheck`, `checkstyleMain`,
`checkstyleTest` and `:iceberg-kafka-connect:iceberg-kafka-connect:test` (167
tests) pass; the `kafka-connect-runtime` integration tests (Docker) were not ru
n.
---
**AI Disclosure**
- Model: Claude Opus 5.5
- Platform/Tool: Claude Code
- Human Oversight: fully reviewed
- Prompt Summary: Model the Kafka Connect sink's commit protocol in TLA+
with the Specula workflow, reproduce violations against released jars, and
prepare a minimal fix with a regression test. The fix, the test and this
description are generated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
