grantatspothero commented on PR #10523: URL: https://github.com/apache/iceberg/pull/10523#issuecomment-2174646888
Found a problem with the approach. This assumption is incorrect for all `SnapshotProducer` > no orphaned manifests could exist if no retries have occurred. This is incorrect for `MergingSnapshotProducer` which merges manifests by writing the unmerged manifests, creating a new merged manifest, and marking the unmerged manifests for deletion. You can see this in [MergingSnapshotProducer.apply()](https://github.com/apache/iceberg/blob/apache-iceberg-1.5.2/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L840-L853) My proposal is we require `SnapshotProducer` to opt-in to this optimization with a method like `protected boolean canSkipCleanupAfterCommitSuccess()` which defaults to false. Suggestion is only latency sensitive operations like `FastAppend` would override this method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
