Guosmilesmile commented on code in PR #17113:
URL: https://github.com/apache/iceberg/pull/17113#discussion_r3543087016
##########
docs/docs/flink-maintenance.md:
##########
@@ -82,6 +82,49 @@ Used to remove files which are not referenced in any
metadata files of an Iceber
.deleteBatchSize(1000))
```
+#### ConvertEqualityDeletes
+Converts equality delete files staged on a source branch into deletion vectors
(DVs) on the target branch. Requires table format version >= 3 (which
introduces DVs). This is typically used together with the Flink `IcebergSink`,
which writes new data files and equality deletes to a staging branch; the
converter then resolves the equality deletes into row-position DVs and commits
the data files together with the DVs to the target branch.
+
+The pipeline runs a single conversion cycle for every trigger event.
Internally it is split into parallel stages (planner → reader → PK index → DV
writer → committer), and it uses the same maintenance-framework lock as other
tasks to guarantee mutual exclusion with concurrent maintenance operations
(e.g. compaction) on the same table.
+
+```java
+.add(ConvertEqualityDeletes.builder()
+ .stagingBranch("staging")
+ .equalityFieldColumns(ImmutableList.of("id"))
+ .scheduleOnEqDeleteFileCount(10)
+ .parallelism(4))
+```
+
+Notes:
+
+**Prerequisites and staging-branch layout**
+
+- Table format version must be `>= 3` (the version that introduces deletion
vectors). This is validated at job startup so a wrongly-configured job fails
fast instead of silently doing nothing.
+- The staging branch is expected to be produced by the Flink `IcebergSink` in
V3 mode: it may contain new data files, equality delete files, and deletion
vectors, but **must not** contain V2 positional delete files. The planner
throws if it encounters any V2 positional delete on the staging branch.
Review Comment:
Right, fix it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
