dramaticlly commented on code in PR #18477:
URL: https://github.com/apache/iceberg/pull/18477#discussion_r4519541248
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java:
##########
@@ -407,14 +410,29 @@ public long totalFilesCount() {
private static class ReadManifest implements
FlatMapFunction<ManifestFileBean, FileInfo> {
private final Broadcast<Table> table;
+ private final SetAccumulator<String> missingManifests;
- ReadManifest(Broadcast<Table> table) {
+ ReadManifest(Broadcast<Table> table, SetAccumulator<String>
missingManifests) {
Review Comment:
@RussellSpitzer I think you suggestion works great 👍 and now it reads much
easier.
As a follow up question, I am wondering if the special extension of
ReadManifest shall exists in ExpireSnapshotsAction only or it shall be
generically applied to the all spark actions.
Like if there's manifests missing on disk, shall all other spark based
maintenance action like rewrite manifests, orphan removal to fail loud or can
follow the similar best effort approach here? To me, ReadManifest seems like a
mapping function to translate the manifest to a file info, so ability to handle
IO problems such as missing input can be generic bundled? If there's concern on
changing ReadManifest, maybe can move `SkipMissingManifest` to BaseSparkAction
as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]