laskoviymishka opened a new pull request, #947: URL: https://github.com/apache/iceberg-go/pull/947
Closes #946. Sustained CDC workloads (Postgres → Iceberg replication, ~one snapshot per few seconds with one eq-delete per UPDATE/DELETE) accumulate equality-delete files in every snapshot's manifests forever. After compaction, every preexisting eq-delete is provably dead — no surviving applicable data file has seq < eq-delete.seq — but the old RewriteDataFiles only removed position deletes, leaving eq-deletes in place. Manifest fanout grew linearly with the CDC duration. This change adds three pieces: 1. table/compaction.DecideDeadEqualityDeletes — a pure predicate that exactly matches the v2 reader (table/scanner.go matchEqualityDeletesToData): E applies to D iff E.seq > D.seq AND (either side has empty partition OR partition tuples match). SpecID is intentionally NOT in the predicate — using it would silently drop eq-deletes the reader still applies under partition-spec evolution. 2. table/compaction.CollectDeadEqualityDeletes — walks the snapshot's manifests via the existing public Snapshot.Manifests + FetchEntries APIs, builds a SurvivorSurvey, and returns the dead set. Two-pass design: skip the data-manifest walk if there are no eq-delete candidates. 3. table.RewriteDataFilesOptions.ExtraDeleteFilesToRemove — a new field on the options struct that the executor passes through to ReplaceFiles. The caller (cmd/iceberg/compact.go or any library user) computes dead eq-deletes via CollectDeadEqualityDeletes against the same snapshot and threads them in. The executor itself is policy-free. The CLI gains a --preserve-dead-equality-deletes flag (default off, i.e., cleanup is on by default — recommended for sustained CDC). The RewriteDataFiles signature collapses (partialProgress, snapshotProps, opts...) into a single RewriteDataFilesOptions struct. RewriteResult splits RemovedDeleteFiles into RemovedPositionDeleteFiles + RemovedEqualityDeleteFiles for observability. Tests: - table/compaction/eq_delete_decision_test.go pins the predicate against the scanner's rule across mixed-partition / cross-spec / boundary cases. - table/compaction/cdc_stress_test.go runs 80 CDC cycles (81 data files + 80 eq-deletes pre-compaction → 1 data file + 0 eq-deletes post). Skipped under -short. - table/rewrite_data_files_test.go covers full-rewrite drops, partial-rewrite preserves, partial-progress preserves (per-group cleanup is a follow-up), and the CLI opt-out path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
