zhengxiang378928908-code opened a new issue, #18843: URL: https://github.com/apache/iceberg/issues/18843
With `overwrite(true)`, `DynamicCommitter.replacePartitions` adds only the data files of each `WriteResult` to the `ReplacePartitions` operation ([DynamicCommitter.java#L285-L298](https://github.com/apache/iceberg/blob/5b99042b4/flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicCommitter.java#L285-L298)). Delete files produced by upserts or CDC deletes are omitted from the overwrite commit without being rejected, which can leave rows visible that those deletes should remove. The [Flink docs](https://github.com/apache/iceberg/blob/5b99042b4/docs/docs/flink-writes.md#L143) state that OVERWRITE and UPSERT are mutually exclusive, and `IcebergSink` and `FlinkSink` enforce this when the job is built; [`IcebergCommitter`](https://github.com/apache/iceberg/blob/5b99042b4/flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergCommitter.java#L250-L251) also fails with `Cannot overwrite partitions with delete files.` The dynamic sink has neither check. The same overwrite commit path exists in Flink 1.20, 2.2 and 2.3. Reproduction on `main` at `5b99042b4` (Flink 2.3, `TestDynamicIcebergSink` style): write the same row four times to an unpartitioned table with `upsertMode=true`, equality field `id`, and `overwrite(true)`. In the observed run the job succeeds and reading the table returns 3 identical rows. The final overwrite replaced an earlier one-record data file with a three-record file, so the exact count depends on checkpoint timing. The last snapshot has `replace-partitions=true` and `total-delete-files=0`. Expected: reject overwrite results that contain delete files, as the other sinks do, instead of committing only their data files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
