alessandro-nori opened a new pull request, #3064: URL: https://github.com/apache/iceberg-rust/pull/3064
## Which issue does this PR close? Follows up on #2591, which landed `ExpireSnapshotsAction` and called this out as remaining work: "Left for follow-ups: ... `cleanExpiredMetadata` (removing unreferenced partition specs and schemas)." ## What changes are included in this PR? Adds `clean_expired_metadata(bool)` to `ExpireSnapshotsAction` (off by default). When enabled, the action also emits `RemoveSchemas` / `RemovePartitionSpecs` for partition specs and schemas that no surviving snapshot references anymore. - **Schemas** need no I/O: each snapshot records its `schema_id`, so the reachable set is a union over surviving snapshots, seeded with the current schema. - **Partition specs** require reading each surviving snapshot's manifest list to collect `partition_spec_id`s. The walk is skipped entirely when the table has at most one spec (the common case, so no I/O is added there), reads run concurrently via `buffer_unordered`, and the loop breaks early once every spec is proven reachable. - The current schema and default spec are always seeded as reachable, so they are never removed. - `commit()` no longer returns early when nothing expires, since metadata cleanup can still have work to do; it falls back to an empty commit only when no updates are produced at all. Two intentional differences from the Java implementation, both toward safety/precision: 1. Reachability is computed over the snapshots that **survive** the expiry, rather than over the retained set. Java walks `idsToRetain`, which can still contain a snapshot named explicitly for expiry, causing it to over-retain specs in that case. 2. A surviving snapshot with no `schema_id` suppresses schema removal entirely. Java treats an unknown schema as referencing nothing, which can drop a schema that is still in use; since the id is an explicit `Option` here, this takes the conservative path. ## Are these changes tested? Yes — 9 new unit tests: unreferenced schema/spec removed, current schema and default spec never removed, schema/spec still used by a retained snapshot survives, the unknown-`schema_id` case, the flag being off by default, cleanup running when nothing expires, and the single-spec table skipping manifest reads. The spec tests write real manifest lists into the table's in-memory `FileIO`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
