mxm opened a new pull request, #18674:
URL: https://github.com/apache/iceberg/pull/18674
### Summary
To avoid rebuilding the index for every trigger, the planner has logic to
reuse the
PK index between triggers. This PR fixes cases where the index can be
incomplete, even
though the planner would consider the index fresh.
These cases are:
1. An index update fails on a trigger with nothing to convert. There are no
retries,
because only triggers that convert a staging snapshot are planned again.
2. A job restores from a checkpoint taken before a conversion commit. The
index
does not have the rows of that commit, and commits by other writers before
it
are missed.
3. A rebuild fails half way in the planner and is still marked as done.
4. A conversion fails while the target branch has no snapshot yet. The retry
resolves its equality deletes against the rows the failed attempt indexed.
### Constraints
All of these issues stem from two constraints of ConvertEqualityDeletes:
A. Conversion commits are made independently of the Flink checkpointing
interval
B. Any failures in the conversion process must not fail the Flink job
### Changes in this PR
Now the planner checks the target branch before it uses the index. The index
is
used only if the head of the target branch is the snapshot the index was
built
at, or the conversion commit of the last planned staging snapshot directly on
top of it. In every other case the planner rebuilds the index.
A rebuild forgets the indexed snapshot first and records the new one only
after
it has emitted all read commands. A rebuild that fails half way is therefore
repeated, also after a restore. While the target branch has no snapshot,
every
trigger starts a new index generation, a counter that goes up with each
rebuild. The workers drop the rows indexed so far when the generation
changes.
If an index update fails in a reader or worker, the committer keeps the
failed
generation in its checkpointed state and does not commit a conversion on it.
The next trigger plans the same staging snapshot again on a rebuilt index.
On a
trigger with nothing to convert, this wastes one conversion unless the index
is
rebuilt in between. We accept that, because we do not want to fail the job.
---
Developed with AI assistance. Every change was fully reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
