raunaqmorarka commented on code in PR #18074:
URL: https://github.com/apache/iceberg/pull/18074#discussion_r4449923613
##########
core/src/main/java/org/apache/iceberg/BaseTransaction.java:
##########
@@ -455,17 +455,31 @@ private void applyUpdates(TableOperations underlyingOps) {
}
// returns the manifest lists and manifests referenced by the given
committed snapshots
- private static Set<String> committedFiles(FileIO io, Set<Snapshot>
snapshots) {
+ private Set<String> committedFiles(FileIO io, Set<Snapshot> snapshots) {
Set<String> committedFiles = Sets.newHashSet();
for (Snapshot snap : snapshots) {
committedFiles.add(snap.manifestListLocation());
- snap.allManifests(io).forEach(manifest ->
committedFiles.add(manifest.path()));
+ committedFiles.addAll(committedManifestPaths(io, snap));
}
return committedFiles;
}
+ // uses the manifest paths held by the update that wrote the snapshot's
manifest list, if any
+ private Set<String> committedManifestPaths(FileIO io, Snapshot snapshot) {
Review Comment:
Removed it from both helpers.
##########
core/src/main/java/org/apache/iceberg/SnapshotProducer.java:
##########
@@ -108,7 +109,10 @@ public void accept(String file) {
private final String commitUUID = UUID.randomUUID().toString();
private final AtomicInteger manifestCount = new AtomicInteger(0);
private final AtomicInteger attempt = new AtomicInteger(0);
- private final List<String> manifestLists = Lists.newArrayList();
+
+ /** Manifest paths written to each manifest list, keyed by manifest list
location. */
+ private final Map<String, Set<String>> manifestPathsByManifestList =
Maps.newLinkedHashMap();
Review Comment:
Switched to `HashMap`.
##########
core/src/main/java/org/apache/iceberg/SnapshotProducer.java:
##########
@@ -88,7 +89,7 @@
abstract class SnapshotProducer<ThisT> implements SnapshotUpdate<ThisT> {
private static final Logger LOG =
LoggerFactory.getLogger(SnapshotProducer.class);
static final int MIN_FILE_GROUP_SIZE = 10_000;
- static final Set<ManifestFile> EMPTY_SET = Sets.newHashSet();
+ static final Set<String> EMPTY_SET = Sets.newHashSet();
Review Comment:
Done in a separate first commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
