dhruvarya-db opened a new pull request, #3038: URL: https://github.com/apache/iceberg-rust/pull/3038
The metadata side of expire-snapshots is already in place from three merged PRs: #2591 added the `ExpireSnapshotsAction`, #2664 made it honour the `history.expire.*` table properties by default, and #2667 removed statistics-file metadata for the expired snapshots. Those all edit `metadata.json` — dropping snapshots and their statistics — but none of them delete anything from storage, so the manifest lists, manifests, and data files that only the expired snapshots referenced are left behind. This PR adds the detection step that finds those files, without deleting anything. `iceberg::transaction::unreferenced_files(table, expired_snapshot_ids)` reads the manifest lists and manifests of the expiring snapshots and returns the files reachable only from them, grouped by kind (manifest lists, manifests, data files, delete files, and statistics). It mirrors Java's `ReachableFileCleanup`: the result is the reference-count difference `files(expired) - files(retained)`, so anything a surviving snapshot still points at is never returned. Content files (data and deletes) are only collected when `gc.enabled` is set, matching `drop_table_data`, since they can be shared across tables; the table-private metadata is always collected. Manifests are read through `ManifestReader`, so encrypted manifests are handled transparently. Any read failure aborts the whole call: a manifest list or manifest that can't be read — for a retained or an expired snapshot alike — leaves the reachable set incomplete, and continuing could delete a file that is still live. This matches Java's `ReachableFileCleanup`. This is metadata analysis only — nothing is deleted here. A follow-up PR will wire this into the commit path and perform the actual file deletion. Part of #2145. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
