mengna-lin opened a new pull request, #18477: URL: https://github.com/apache/iceberg/pull/18477
ExpireSnapshotsSparkAction reads every manifest list and manifest reachable from the expired snapshots to compute files to delete. If any is already gone the read throws NotFoundException and the action fails before deleting anything, so expiration can never make progress and orphans keep accumulating. This adds an opt-in best-effort-on-missing-metadata option (default false). When enabled, metadata reachable only from expired snapshots that is missing is skipped, and the skipped locations are surfaced as one NotFoundException after every fully-resolved file is deleted. Default behavior is unchanged. Issue: https://github.com/apache/iceberg/issues/17715 Notes for reviewers - Default unchanged — with the option off, snapshots resolve through the existing ReadManifest fail-fast seam; no pre-scan runs. - Safety — under-deletes only; metadata reachable from a retained snapshot still fails fast. - Two failure points — a missing manifest list is caught up front via InputFile.exists(); a manifest lost mid-read via BestEffortManifestIterator. Both merge into one NotFoundException. - Tests — best-effort skip and default fail-fast for a missing manifest list and a missing manifest, plus fail-fast when reachable from a retained snapshot. This is a port of #17757 from Spark 4.1 to Spark 4.2; the code is identical modulo the module path. --- **AI Disclosure** - Model: Claude Opus 4.8 - Platform/Tool: Claude Code - Human Oversight: partially reviewed - Prompt Summary: Port PR #17757 (opt-in best-effort ExpireSnapshots on missing metadata) from the Spark 4.1 module to the Spark 4.2 module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
