max-parke-scale opened a new pull request, #4167: URL: https://github.com/apache/iceberg-python/pull/4167
Closes #4165 # Rationale for this change `_OverwriteFiles._existing_manifests` keeps every parent manifest that has no live files. It reads each manifest with `discard_deleted=True`, so a manifest whose entries are all DELETED comes back empty and takes the "nothing to delete" branch. Each overwrite also writes a manifest for its own deletions, so with explicit `delete_data_file` overwrites the manifest list grows by one per commit and every later commit and scan opens all of them. In one production table this reached 388 manifests, 387 of them deleted-only. This skips parent manifests with no added or existing files before reading them, the rule Java applies in [`MergingSnapshotProducer.apply`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java). Earlier snapshots keep their own manifest lists, so time travel is unaffected, and each new snapshot still lists the manifest recording its own deletions. ## Are these changes tested? Yes. `test_overwrite_drops_manifests_without_live_files` runs three overwrites and checks that each snapshot lists two manifests and that the first snapshot still reads its own row; on `main` the counts are `[2, 3, 4]`. The unit suite and `prek` pass. ## Are there any user-facing changes? No API change. Tables written with explicit-delete overwrites stop accumulating deleted-only manifests, and existing tables shed them on their next overwrite. 🧑💻🤖 — posted via [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
