MavsLee opened a new issue, #12200: URL: https://github.com/apache/iceberg/issues/12200
### Apache Iceberg version 1.7.1 (latest release) ### Query engine Other ### Please describe the bug 🐞 When using Iceberg Java SDK `org.apache.iceberg.RemoveSnapshots`to expire snapshots on table with multiple references(i.e branch/tag), when different branches/tag have different retention time set, expire snapshot operation can not fully clean up data files that referenced by old snapshots. For example, a table has multiple snapshot S1(tag t1), S2(branch v1), S3 First branch b1 is expired, then tag t1 expired, data in S1 is expected to be deleted. But it is not. Root cause is with currently implementation, it uses `IncrementalFileCleanup` strategy to calculate expired data files when refs count is 1, so it can't trace back to Snapshot 1, because Snapshot S3 ancestor's snapshot S2 already expired. To fix, I think it should do reachability analysis by using `ReachableFileCleanup`. I plan to have a UT to repro the issue and update `org.apache.iceberg.RemoveSnapshots#cleanExpiredSnapshots` to use ReachableFileCleanup when existing snapshots are not consecutive. Let me know if you have any questions about the issue or how to fix it. BTW, I checked spark `ExpireSnapshotsSparkAction` doesn't rely on the SDK code to calculate expired file, it has different implementation `org.apache.iceberg.spark.actions.ExpireSnapshotsSparkAction#expireFiles` which always use reachability analysis. So it doesn't have such issue. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
