wgtmac opened a new pull request, #1104:
URL: https://github.com/apache/iceberg-cpp/pull/1104
### Problem
Transaction replay currently calls `CleanupUpdates()` before reapplying
pending updates. This deletes staged files that later updates in the same
transaction may still reference.
For example:
1. A staged append creates manifest M0.
2. A later `RewriteManifests` replaces M0 with a caller-provided manifest.
3. An unrelated property update forces the transaction to replay.
4. Cleanup deletes M0, and the append generates a different manifest path.
5. The rewrite fails because its declared source manifest no longer exists.
Simply retaining all cached state is also unsafe. A manifest can keep the
same path while inheriting different sequence numbers or row IDs on replay.
Derived caches and explicit replacements must not silently carry values
from an earlier attempt into the committed snapshot.
### Solution
Separate replay preparation from terminal file cleanup:
- Add `PrepareForReplay()` to reset attempt-dependent state without deleting
staged files. FastAppend and MergingSnapshotUpdate reuse their immutable
append outputs. MergingSnapshotUpdate clears derived filter/merge caches,
and RewriteManifests rebuilds its rewrite outputs.
- Validate the sequence number, minimum sequence number, and first row ID of
explicitly replaced source manifests. Changed inheritance metadata causes
a validation failure.
- Preserve copied-manifest summaries across attempts. Resume incomplete
copy batches from the first uncopied input, and merge each copy's summary
only after that copy succeeds.
- Centralize generated-file cleanup in SnapshotUpdate's staging registry,
including owned merged-DV files. After success, protect files referenced
by every retained snapshot added by the transaction, including intermediate
and staged-only snapshots.
- Derive expected snapshot IDs from the final `AddSnapshot` changes. Skip
snapshot-file cleanup if committed metadata omits any expected snapshot
or file discovery fails, so a partial keep-set cannot delete live files.
Known failures and aborts clean staged files; unknown outcomes preserve
them.
- Reset per-attempt reporting snapshots and expiration cleanup plans.
A cherry-pick that becomes a fast-forward must not report an abandoned
draft, and a no-op expiration must not execute an earlier cleanup plan.
Redundant cleanup helpers and file tracking are removed, and the cleanup
contract now explicitly leaves physical deletion to the staging registry.
### Validation
Regression coverage includes repeated append-to-rewrite replay, inherited
metadata changes across branches, stale replacement rejection, copied
manifest reuse and copy resumption, intermediate snapshot retention, and
cherry-pick transitions to fast-forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
