anoopj commented on code in PR #15653:
URL: https://github.com/apache/iceberg/pull/15653#discussion_r2991907491
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -866,6 +871,38 @@ private void validateAddedDVs(
}
}
+ private Iterable<ManifestFile> filterManifestsByPartition(
+ TableMetadata base, Expression conflictDetectionFilter,
List<ManifestFile> manifests) {
+ if (conflictDetectionFilter == null || conflictDetectionFilter ==
Expressions.alwaysTrue()) {
+ return manifests;
+ }
+
+ // if any concurrent manifest was written with a different partition spec,
skip pruning
+ // to avoid incorrectly excluding manifests when a spec change happened
during validation
+ int defaultSpecId = base.defaultSpecId();
+ if (manifests.stream().anyMatch(m -> m.partitionSpecId() !=
defaultSpecId)) {
+ return manifests;
+ }
+
+ Map<Integer, PartitionSpec> specsById = base.specsById();
+ Map<Integer, ManifestEvaluator> evaluators = Maps.newHashMap();
+ return Iterables.filter(
Review Comment:
That is a good idea. Will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
