mxm commented on code in PR #18717:
URL: https://github.com/apache/iceberg/pull/18717#discussion_r4479060005
##########
docs/docs/flink-maintenance.md:
##########
@@ -115,12 +115,14 @@ Notes:
- The PK-index worker keeps a keyed row-position index of the target branch in
Flink state, persisted with each Flink checkpoint. Its size scales with the
number of live rows in the table for the configured equality columns; size
Flink state backend, checkpoint storage, and TaskManager memory accordingly.
**RocksDB is the preferred state backend** because the PK index can grow beyond
the available heap and RocksDB spills to local disk instead of failing with an
`OutOfMemoryError`.
- The index is **reused across triggers and restored on failover** while it
matches the head of the target branch: the head is the snapshot the index was
built from, or the converter's own commit for the last planned staging snapshot
directly on top of it. Each cycle applies its staging snapshot's changes to the
index, so consecutive conversions do not rebuild it.
Review Comment:
Yes, we can refine this now. The index is also reused / refreshed when its
corresponding snapshot is derivable from the head of the target branch. Updated.
##########
docs/docs/flink-maintenance.md:
##########
@@ -115,12 +115,14 @@ Notes:
- The PK-index worker keeps a keyed row-position index of the target branch in
Flink state, persisted with each Flink checkpoint. Its size scales with the
number of live rows in the table for the configured equality columns; size
Flink state backend, checkpoint storage, and TaskManager memory accordingly.
**RocksDB is the preferred state backend** because the PK index can grow beyond
the available heap and RocksDB spills to local disk instead of failing with an
`OutOfMemoryError`.
- The index is **reused across triggers and restored on failover** while it
matches the head of the target branch: the head is the snapshot the index was
built from, or the converter's own commit for the last planned staging snapshot
directly on top of it. Each cycle applies its staging snapshot's changes to the
index, so consecutive conversions do not rebuild it.
-- Otherwise the next trigger **rebuilds the index** from the target branch
head, reading the equality columns of every live data file. This happens:
+- External commits on the target branch (e.g. compaction or direct writes) are
applied to the index as a delta: the planner reads the data files they added
and removed, and the worker indexes or evicts their rows. A file that is added
and removed within the delta is skipped.
+- In the following cases the next trigger **rebuilds the index** from the
target branch head instead, reading the equality columns of every live data
file:
- on cold start without a checkpoint, and on the first trigger after
restoring state written by an older Iceberg version
- - after any other commit on the target branch, e.g. compaction, external
writes, rollback, or replace-main (see *Concurrent commits on the target
branch* below)
+ - when the indexed snapshot is no longer in the history of the target
head, e.g. after rollback, replace-main, or snapshot expiration
- after restoring a checkpoint that predates the latest conversion commit
- when the last planned conversion did not commit
- when an earlier rebuild did not finish
Review Comment:
Applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
