viirya opened a new pull request, #3582: URL: https://github.com/apache/iceberg-rust/pull/3582
## Which issue does this PR close? - Closes #3581. ## What changes are included in this PR? Adds the `write.metadata.delete-after-commit.enabled` table property (default `false`) and a shared `iceberg::delete_removed_metadata_files` helper that matches Java's `CatalogUtil.deleteRemovedMetadataFiles`. Catalogs call it only after the metadata pointer swap succeeds, so a failed commit, or one whose outcome is unknown, never deletes anything. The helper reads the property from the new metadata (as Java does) and deletes the files in the base `metadata_log` that are not in the new one. It diffs the two logs rather than using `TableMetadataBuildResult::expired_metadata_logs`, because `TableCommit::apply` builds the metadata and drops that result before catalogs see it, and the diff is what Java does. Deletion is best-effort: each failure is logged with `tracing::warn!` and the commit still succeeds. Wired into the memory, SQL and Glue catalogs. Not wired into REST, where the server owns metadata files (Java doesn't delete client-side either), or S3 Tables, where the service manages file cleanup. HMS doesn't support `update_table` yet. ## Are these changes tested? Yes. - Memory catalog unit tests: enabled deletes exactly the files that fell out of the log and keeps the ones still in it; unset or `false` keeps everything; a delete that fails (a non-empty directory in place of the old file on the local FS) doesn't fail the commit, and later commits keep deleting. - `table_suite` in `crates/catalog/loader`: the same enabled scenario for the Glue, SQL and memory catalogs. I ran the Glue case locally against the moto and MinIO containers. ## AI Disclosure Written with an AI assistant (Claude). Please check: - The catalog split: wired into memory, SQL and Glue; skipped for S3 Tables even though its client writes the metadata file itself, on the assumption that the service owns cleanup of table bucket files. If S3 Tables should delete them client-side, it's a two-line addition. - The removed-file semantics: diffing base and new `metadata_log` by file path, and reading the property from the new metadata. - The memory catalog releases its state lock before deleting, so deletes don't block other commits. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
