manuzhang commented on code in PR #14264:
URL: https://github.com/apache/iceberg/pull/14264#discussion_r4078728462
##########
core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java:
##########
@@ -133,13 +270,851 @@ private static Map<Long, Integer>
computeSnapshotOrdinals(Deque<Snapshot> snapsh
return snapshotOrdinals;
}
+ /**
+ * Builds a delete file index for existing deletes that were present before
the start snapshot.
+ * These deletes should be applied to data files but should not generate
DELETE changelog rows.
+ * Manifests that cannot hold a delete relevant to the range are pruned
before the index builder
+ * reads the rest, and files outside the affected scope are skipped as
entries are read.
+ */
+ private DeleteFileIndex buildExistingDeleteIndex(
+ Long fromSnapshotIdExclusive, ExistingDeleteScope scope) {
+ if (fromSnapshotIdExclusive == null) {
+ return DeleteFileIndex.emptyIndex();
+ }
+ Snapshot fromSnapshot = table().snapshot(fromSnapshotIdExclusive);
+ Preconditions.checkState(
+ fromSnapshot != null, "Cannot find starting snapshot: %s",
fromSnapshotIdExclusive);
+
+ List<ManifestFile> deleteManifests =
fromSnapshot.deleteManifests(table().io());
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // Prune manifests that cannot contain deletes for any partition affected
by the scan range
+ if (scope != null) {
+ deleteManifests =
+ pruneManifestsByAffectedPartitions(deleteManifests,
scope.allAffectedPartitions);
+ }
+
+ if (deleteManifests.isEmpty()) {
+ return DeleteFileIndex.emptyIndex();
+ }
+
+ // filterData prunes manifests and entries against the scan filter with
per-spec cached
+ // evaluators and keeps minimal stats. Entry pruning also runs the filter
against each delete
+ // file's own stats, which may drop a delete file whose bounds cannot
match the filter; that is
+ // only safe because the task residual re-applies the filter to the rows a
task emits, so
+ // ignoreResiduals must be forwarded (as ManifestGroup does) to disable it
when the residual is
+ // dropped. Deliberately no planWith: this may run lazily under the
cumulative-index monitor on
+ // shared worker-pool threads, and nested submission to the same pool can
starve it
+ DeleteFileIndex.Builder builder =
+ DeleteFileIndex.builderFor(table().io(), deleteManifests)
+ .specsById(table().specs())
Review Comment:
` .schemasById(table.schemas())` is needed as well to cover columns used by
equality deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
