mxm opened a new pull request, #18717:
URL: https://github.com/apache/iceberg/pull/18717
### What to review
This change is based on #18674. The commit to review in this PR is
e982ec35a005f826e99ed167e5be1a543c5febe6.
### Summary
An external commit on the target branch, typically a compaction, makes the
next
trigger clear the PK index and read every data file of the target again. A
compaction typically rewrites a small part of the table, but the rebuild
reads
and writes all data.
With this PR, the planner applies only the data file changes of the external
commits since the indexed snapshot. The index adds the rows of every added
file
and evicts the rows of every removed file. A file that is added and removed
within the range is skipped.
The planner still rebuilds the index when the incremental path does not
work, in
the following cases:
- There is no index yet.
- The indexed snapshot is no longer in the history of the target head
(rollback,
replace-main, snapshot expiration).
- A planned staging snapshot has no conversion commit. Its eq deletes
consumed
index entries, which a delta cannot restore.
- The history has a conversion commit for another staging snapshot, for
example
after a restore from a checkpoint taken before it.
- The delta reads more rows than a rebuild. The planner compares the
manifest list
row counts of the external commits with those of the target head. A
manifest
without row counts also rebuilds.
---
Developed with AI assistance. Every change was fully reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
