moomindani opened a new pull request, #3631: URL: https://github.com/apache/iceberg-python/pull/3631
Closes #3629 # Rationale for this change Adds `table.maintenance.rewrite_manifests()`, the explicit maintenance counterpart of Java's `Table.rewriteManifests()` (Spark: `rewrite_manifests`). When a table accumulates many small manifests from repeated appends, scan planning slows down; this merges the current snapshot's data manifests into fewer manifests sized by `commit.manifest.target-size-bytes` (default 8 MB). - Entries are rewritten as EXISTING and keep their sequence numbers and file sequence numbers. - Delete manifests, and data manifests that need no merging, are kept as-is. - The result is committed as a `replace` snapshot; snapshot totals carry over unchanged (`update_snapshot_summaries` now handles `Operation.REPLACE`), and the summary reports `manifests-created` / `manifests-kept` / `manifests-replaced` / `entries-processed`, matching Java. - V3 tables are rejected with a clear error for now: rewriting must preserve the `first-row-id` of rewritten manifests, which needs the read side of row lineage (#3621). The writer-side plumbing already exists in #3624. Related: #270 tracks automatic manifest merging on write; this PR covers only the explicit maintenance API. **Draft** while the API shape settles: #3124 is introducing `table.maintenance.compact()` on the same `MaintenanceTable`, and I'd like this to land consistently with it. ## Are these changes tested? Yes, five tests using an in-memory catalog with real appends: merging three manifests into one (entry counts, EXISTING status, data unchanged, replace summary with carried-over totals), sequence-number preservation including the merged manifest's `min_sequence_number`, single-manifest no-op (kept as-is, not rewritten), target-size grouping, and `rewrites_needed()`. All fail without the implementation. ## Are there any user-facing changes? Yes: a new `table.maintenance.rewrite_manifests()` API. This pull request and its description were written by Claude Fable 5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
