viirya opened a new issue, #3581: URL: https://github.com/apache/iceberg-rust/issues/3581
### Is your feature request related to a problem or challenge? iceberg-rust does not support the `write.metadata.delete-after-commit.enabled` table property, so old `*.metadata.json` files are never cleaned up after commits. `TableMetadataBuilder::build()` already trims `metadata_log` to `write.metadata.previous-versions-max` and returns the trimmed entries in `TableMetadataBuildResult::expired_metadata_logs`, but nothing reads that field. The files that fall out of the log stay in storage forever, even though no table metadata references them anymore. In Java, `BaseMetastoreTableOperations.commit` and `HadoopTableOperations.commit` call `CatalogUtil.deleteRemovedMetadataFiles(io, base, metadata)` after a successful commit. When `write.metadata.delete-after-commit.enabled` (default `false`) is set on the new metadata, it deletes `base.previousFiles()` minus `metadata.previousFiles()`, best-effort: failures are logged and do not fail the commit. The REST catalog does not do this on the client, since the server owns the metadata files. ### Describe the solution you'd like - Add `write.metadata.delete-after-commit.enabled` to `TableProperties`. - Add a shared helper that matches `CatalogUtil.deleteRemovedMetadataFiles`, and call it after a successful commit in the catalogs that write metadata files themselves (memory, SQL, Glue). - Never delete when the commit fails or its outcome is unknown. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
