RussellSpitzer commented on code in PR #18477:
URL: https://github.com/apache/iceberg/pull/18477#discussion_r4527656022
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java:
##########
@@ -407,14 +410,29 @@ public long totalFilesCount() {
private static class ReadManifest implements
FlatMapFunction<ManifestFileBean, FileInfo> {
private final Broadcast<Table> table;
+ private final SetAccumulator<String> missingManifests;
- ReadManifest(Broadcast<Table> table) {
+ ReadManifest(Broadcast<Table> table, SetAccumulator<String>
missingManifests) {
Review Comment:
I left this as a comment in another thread, but expire snapshots is a
different beast from all the other actions. It's about removing history, and to
accomplish that the only thing we *really* need to do is erase the snapshots
from the metadata.json file. The other part of the action, cleaning up files,
is not really critical to success. If we don't clean up some files that we
otherwise should have, it's not a huge deal.
So basically we're doing the same thing here we were always doing: if we
find some file we're supposed to delete and it's not there, we log a warning
and move on.
For every other action, failing to read a manifest means the table is
corrupted. If I try to rewrite a table or its manifests and a manifest isn't
there, that's a failure, and a "partial success" still leaves us with a corrupt
table. Orphan removal is even worse: it uses the manifests of live snapshots to
decide what's still referenced, so skipping a missing manifest would make its
data files look like orphans and we'd delete them. That turns a repairable
table into permanent data loss. This is also why the valid-files side of expire
stays strict in this PR.
So I'd keep `SkipMissingManifest` private to the expire action rather than
moving it to `BaseSparkAction`. For the corrupted-table cases, I think we need
to finish up a "repair tables" sort of thing instead. If someone has another
use case later we can always refactor then. I tend to be a yngni (your not
gonna need it) reviewer so with only 1 viable usecase I want to keep it tight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]