dramaticlly commented on code in PR #18477:
URL: https://github.com/apache/iceberg/pull/18477#discussion_r4519541248


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java:
##########
@@ -407,14 +410,29 @@ public long totalFilesCount() {
 
   private static class ReadManifest implements 
FlatMapFunction<ManifestFileBean, FileInfo> {
     private final Broadcast<Table> table;
+    private final SetAccumulator<String> missingManifests;
 
-    ReadManifest(Broadcast<Table> table) {
+    ReadManifest(Broadcast<Table> table, SetAccumulator<String> 
missingManifests) {

Review Comment:
   @RussellSpitzer  I think you suggestion works great 👍  and now it reads much 
easier. 
   
   As a follow up question, I am wondering if the special extension of 
ReadManifest shall exists in ExpireSnapshotsAction only or it shall be 
generically applied to the all spark actions. 
   
   Like if there's manifests missing on disk, shall all other spark based 
maintenance action like rewrite manifests, orphan removal to fail loud or can 
follow the similar best effort approach here? To me, ReadManifest seems like a 
mapping function to translate the manifest to a file info, so ability to handle 
IO problems such as missing input can be generic bundled? If there's concern on 
changing ReadManifest, maybe can move `SkipMissingManifest` to BaseSparkAction 
as well?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to