ajanko-shopify opened a new pull request, #18916:
URL: https://github.com/apache/iceberg/pull/18916

   This PR supersedes #17901 by @fmorillo7694. It keeps their original commit, 
rebases it onto `main`, and addresses the open review comment. All credit for 
the fix and the regression test goes to them. If they'd rather continue in 
#17901, I'm happy to close this PR.
   
   ## Problem
   
   `IcebergCommitter` reads `getMaxCommittedCheckpointId()` once, before the 
commit. Another commit for the same checkpoint can land after that read, and 
the second commit then applies cleanly on top of the first. The checkpoint's 
files are committed twice. See #17901 and #14425 for the original analysis and 
the deterministic reproduction.
   
   We also hit this in production with blue/green deployments, where the old 
job keeps running for a short time after the savepoint is taken. The old job 
commits checkpoint N. Seconds later, the new job commits the same committable, 
which it restored from the savepoint, under the same job ID. In one table, we 
found 10 data files that were each added twice, in consecutive snapshots.
   
   ## Changes
   
   The first commit is the original change from #17901, rebased onto `main`. 
The second commit makes these changes:
   
   - **Moves `MaxCommittedCheckpointIdValidator` into `SinkUtil`**, as @mxm 
asked in https://github.com/apache/iceberg/pull/17901#discussion_r4155439525. 
Its mismatch exception moves with it.
   - **Adapts the validator to #18303.** `DynamicCommitter` now identifies its 
own snapshots by sink ID, and `IcebergCommitter` still uses the job ID and 
operator ID. So each caller now passes a predicate that matches its own 
snapshot summaries.
   - **In `IcebergCommitter`, validates only after a restore.** This matches 
the up-front check since #18123. After a stateless start, a job can reuse its 
job ID while its checkpoint IDs start from 1 again. Without this condition, the 
validator would skip those commits and silently drop data. 
`testCommitTxnAfterStatelessRestart` caught this. Duplicate commits come from 
restored committables, so restored jobs, including blue/green deployments, are 
still protected.
   - **Updates the regression test** for the new `IcebergCommitter` constructor.
   
   Like #17901, this targets Flink 2.3 only. Backports can follow once the 
approach is approved.
   
   ## Testing
   
   - `TestIcebergCommitterDuplicateCommit` passes. It fails when the validator 
is disabled, with 2 snapshots instead of 1.
   - `TestIcebergCommitter` (336), `TestIcebergFilesCommitter` (96), 
`TestIcebergSink` (312), `TestDynamicCommitter` (14) and 
`TestDynamicIcebergSink` (50) all pass.
   - `spotlessCheck`, `checkstyleMain` and `checkstyleTest` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to