danielcweeks commented on code in PR #15653:
URL: https://github.com/apache/iceberg/pull/15653#discussion_r2990403109
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -866,6 +871,25 @@ private void validateAddedDVs(
}
}
+ private Iterable<ManifestFile> filterManifestsByPartition(
+ TableMetadata base, Expression conflictDetectionFilter,
List<ManifestFile> manifests) {
+ if (conflictDetectionFilter == null || conflictDetectionFilter ==
Expressions.alwaysTrue()) {
+ return manifests;
+ }
+
+ Map<Integer, PartitionSpec> specsById = base.specsById();
+ return Iterables.filter(
+ manifests,
+ manifest -> {
+ PartitionSpec spec = specsById.get(manifest.partitionSpecId());
+ Expression partitionFilter =
+ Projections.inclusive(spec,
caseSensitive).project(conflictDetectionFilter);
+ ManifestEvaluator evaluator =
+ ManifestEvaluator.forPartitionFilter(partitionFilter, spec,
caseSensitive);
Review Comment:
You might be able to pull this out of the loop and reuse the evaluator.
This is especially true if there's only one spec (common), but pre-creating the
filters would save recreating for every iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
