RussellSpitzer commented on code in PR #18477:
URL: https://github.com/apache/iceberg/pull/18477#discussion_r4518508541
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java:
##########
@@ -145,10 +146,13 @@ protected Table newStaticTable(String
metadataFileLocation, FileIO io) {
}
protected Dataset<FileInfo> contentFileDS(Table table) {
- return contentFileDS(table, null);
+ return contentFileDS(table, null, ReadManifest::new);
}
- protected Dataset<FileInfo> contentFileDS(Table table, Set<Long>
snapshotIds) {
+ protected Dataset<FileInfo> contentFileDS(
+ Table table,
+ Set<Long> snapshotIds,
+ Function<Broadcast<Table>, ReadManifest> readManifestFactory) {
Review Comment:
I think this is in the right direction, but I'd prefer (even if it takes
additional code) to not add a new parameter to this method. I generally think
we shouldn't add Functions anywhere into our methods if we can at all help it.
Here I would try to still keep the split between the implementations
So
Something like
```java
// BaseSparkAction
protected Dataset<FileInfo> contentFileDS(Table table, Set<Long>
snapshotIds) {
return manifestBeanDS(table, snapshotIds)
.flatMap(new ReadManifest(broadcastTable(table)), FileInfo.ENCODER);
}
// ExpireSnapshotsSparkAction
private Dataset<FileInfo> deleteCandidateFileDS(
TableMetadata metadata, Set<Long> deletedSnapshotIds) {
Table staticTable = newStaticTable(metadata, table.io());
Set<Long> resolvableSnapshotIds = resolvableSnapshotIds(metadata,
deletedSnapshotIds);
return manifestBeanDS(staticTable, resolvableSnapshotIds)
.flatMap(new SkipMissingManifest(broadcastTable(staticTable)),
FileInfo.ENCODER)
.union(manifestDS(staticTable, resolvableSnapshotIds))
.union(manifestListDS(staticTable, resolvableSnapshotIds))
.union(statisticsFileDS(staticTable, deletedSnapshotIds));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]