RussellSpitzer commented on code in PR #18477:
URL: https://github.com/apache/iceberg/pull/18477#discussion_r4527656022


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java:
##########
@@ -407,14 +410,29 @@ public long totalFilesCount() {
 
   private static class ReadManifest implements 
FlatMapFunction<ManifestFileBean, FileInfo> {
     private final Broadcast<Table> table;
+    private final SetAccumulator<String> missingManifests;
 
-    ReadManifest(Broadcast<Table> table) {
+    ReadManifest(Broadcast<Table> table, SetAccumulator<String> 
missingManifests) {

Review Comment:
   I left this as a comment in another thread, but expire snapshots is a 
different beast from all the other actions. It's about removing history, and to 
accomplish that the only thing we *really* need to do is erase the snapshots 
from the metadata.json file. The other part of the action, cleaning up files, 
is not really critical to success. If we don't clean up some files that we 
otherwise should have, it's not a huge deal.
   
   So basically we're doing the same thing here we were always doing: if we 
find some file we're supposed to delete and it's not there, we log a warning 
and move on.
   
   For every other action, failing to read a manifest means the table is 
corrupted. If I try to rewrite a table or its manifests and a manifest isn't 
there, that's a failure, and a "partial success" still leaves us with a corrupt 
table. Orphan removal is even worse: it uses the manifests of live snapshots to 
decide what's still referenced, so skipping a missing manifest would make its 
data files look like orphans and we'd delete them. That turns a repairable 
table into permanent data loss. This is also why the valid-files side of expire 
stays strict in this PR.
   
   So I'd keep `SkipMissingManifest` private to the expire action rather than 
moving it to `BaseSparkAction`. For the corrupted-table cases, I think we need 
to finish up a "repair tables" sort of thing instead. If someone has another 
use case later we can always refactor then. I tend to be a yngni (your not 
gonna need it) reviewer so with only 1 viable usecase I want to keep it tight.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to