RussellSpitzer commented on code in PR #18477:
URL: https://github.com/apache/iceberg/pull/18477#discussion_r4518508541


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java:
##########
@@ -145,10 +146,13 @@ protected Table newStaticTable(String 
metadataFileLocation, FileIO io) {
   }
 
   protected Dataset<FileInfo> contentFileDS(Table table) {
-    return contentFileDS(table, null);
+    return contentFileDS(table, null, ReadManifest::new);
   }
 
-  protected Dataset<FileInfo> contentFileDS(Table table, Set<Long> 
snapshotIds) {
+  protected Dataset<FileInfo> contentFileDS(
+      Table table,
+      Set<Long> snapshotIds,
+      Function<Broadcast<Table>, ReadManifest> readManifestFactory) {

Review Comment:
   I think this is in the right direction, but I'd prefer (even if it takes 
additional code) to not add a new parameter to this method. I generally think 
we shouldn't add Functions anywhere into our methods if we can at all help it.
   
   Here I would try to still keep the split between the implementations
   
   So
   
   Something like 
   ```java
   // BaseSparkAction
     protected Dataset<FileInfo> contentFileDS(Table table, Set<Long> 
snapshotIds) {
       return manifestBeanDS(table, snapshotIds)
           .flatMap(new ReadManifest(broadcastTable(table)), FileInfo.ENCODER);
     }
   
   // ExpireSnapshotsSparkAction
     private Dataset<FileInfo> deleteCandidateFileDS(
         TableMetadata metadata, Set<Long> deletedSnapshotIds) {
       Table staticTable = newStaticTable(metadata, table.io());
   
       Set<Long> resolvableSnapshotIds = resolvableSnapshotIds(metadata, 
deletedSnapshotIds);
   
       return manifestBeanDS(staticTable, resolvableSnapshotIds)
           .flatMap(new SkipMissingManifest(broadcastTable(staticTable)), 
FileInfo.ENCODER)
           .union(manifestDS(staticTable, resolvableSnapshotIds))
           .union(manifestListDS(staticTable, resolvableSnapshotIds))
           .union(statisticsFileDS(staticTable, deletedSnapshotIds));
     }
   
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to