krisnaru opened a new pull request, #16220: URL: https://github.com/apache/iceberg/pull/16220
Fixes https://github.com/apache/iceberg/issues/14458 ## Summary - Removes broken snapshotIds.contains(entry.snapshotId()) filter from RewriteTablePathUtil that silently dropped live, un replicated files when their adding snapshot was expired - Replaces snapshot-ID-based entry filtering with entry.isLive() only — all live content files are included in the copy plan - Adds anti-join dedup for incremental copies: filters out previously-replicated files by comparing against the start version's content files via contentFileDS - Fixes both full copy (expired snapshot IDs missing from endMetadata.snapshots()) and incremental copy (expired snapshot IDs missing from deltaSnapshotIds after RewriteManifests + ExpireSnapshots) ## Problem RewriteTablePathUtil.writeDataFileEntry filtered copy plan entries using snapshotIds.contains(entry.snapshotId()). This is fundamentally broken because entry.snapshotId() reflects the snapshot that originally added the file, which can be expired at any time by table maintenance. When RewriteManifests reorganizes entries from expired snapshots into new manifests, the manifest is correctly selected for processing but the entry-level filter drops the files — causing silent data loss at the target. **Scenario**: Append (S3) → RewriteManifests (S4) → Expire S3 → Incremental replication builds deltaSnapshotIds = {S4}, but file C has entry.snapshotId() = S3 → {S4}.contains(S3) == false → file C excluded from copy plan. ## Approach 1. Entry level: Include all live entries in copy plan (no snapshot ID check) 2. Incremental dedup: Anti-join against start version's content files using contentFileDS from BaseSparkAction 3. Manifest selection: Unchanged — manifestsToRewrite still uses deltaSnapshotIds from version history ## Test plan - Verify existing TestRewriteTablePathsAction tests pass (full copy, incremental, expire scenarios) - Verify existing TestRewriteTablePathUtil tests pass - Add integration test: incremental replication after append + RewriteManifests + ExpireSnapshots - Verify incremental copy produces correct file counts (no duplicate copies of already-replicated files) - Verify full copy after many snapshots with expired snapshot IDs includes all live data -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
