raghav-reglobe opened a new pull request, #2755: URL: https://github.com/apache/iceberg-rust/pull/2755
## Summary Adds `maintenance::RemoveOrphanFilesAction`: identify — and, unless dry-run, delete — files under the table location that no retained snapshot or table metadata references. This revives the table-maintenance direction proposed in #1453, modeled on iceberg-go's `Table.DeleteOrphanFiles` (itself modeled on Java's `DeleteOrphanFiles` action). > [!NOTE] > **Stacked on #2754** (`FileIO::list_prefix`) — the first commit here is that PR; the net-new to review is the second commit. Happy to rebase once #2754 lands. ## Safety design (deleting the wrong file is unrecoverable) - **Dry-run first**: `dry_run(true)` produces the full orphan report with zero writes, so callers can verify it (e.g. against an independent engine's `all_files` metadata table) before ever running delete mode. - **Age guard**: only files last-modified before `older_than_ms` (default now − 3 days, the Java/Go default) are candidates — an in-flight commit's freshly-written files are unreferenced until the commit lands. Files with an unknown modification time are never touched. - **Conservative referenced set**: current + historical metadata files, the version hint, statistics + partition statistics, and every retained snapshot's manifest list, manifests, and **all entries' file paths regardless of entry status**. Including deleted-status entries means the action can never remove a file any retained manifest still mentions, and keeps the report directly verifiable against Java's `all_files` metadata table (which also includes logically-deleted files). This is a deliberate, documented divergence from iceberg-go's `discardDeleted` behavior. - **`PrefixMismatchMode`** (Error default / Ignore / Delete) with `equal_schemes` / `equal_authorities`, matching iceberg-go's handling of files referenced under equivalent URI schemes/authorities. - Deletion is bounded-concurrency and best-effort, with per-file failures reported rather than aborting the sweep. ## Validation - URI-normalization unit tests; end-to-end tests on a real table (memory catalog + parquet writer chain): the dry-run report contains exactly the planted garbage and never a referenced file, the age guard protects fresh files, and delete removes only the garbage — the table remains fully readable and a second pass converges to zero. - Full `iceberg` suite green; `cargo fmt` clean; `public-api.txt` regenerated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
