mikebewell opened a new pull request, #18796: URL: https://github.com/apache/iceberg/pull/18796
Flink `INSERT OVERWRITE` on a partitioned table replaces only the partitions that receive new rows, so a batch job that rebuilds a whole table cannot drop a partition the new run no longer produces. This adds a write option `overwrite-mode` with values `dynamic` (default) and `full`, plus a table property `write.flink.overwrite-mode` as the per-table default. In `full` mode the committer commits `OverwriteFiles` with an always-true row filter, which Iceberg evaluates against each manifest's own partition spec, so files written before a partition field was added are replaced too. A `dynamic` hint on a `full` table and `PARTITION (...)` in `full` mode are rejected, an empty full overwrite is skipped, and the dynamic sink rejects `full`. Related: #9644 Test plan: `TestFlinkWriteConf` covers option, Flink config and table property resolution and the rejections. `TestIcebergCommitter` and `TestIcebergFilesCommitter` cover both committers: full overwrite replaces all partitions and a file written under an older spec, dynamic keeps untouched partitions, empty full overwrite adds no snapshot. `TestFlinkTableSink` covers the SQL path for the hint, the property, the older-spec table, and the two rejections. Ran `spotlessApply`, `checkstyleMain`, `checkstyleTest` and these four classes on Flink 2.3, JDK 21: 764 tests, 0 failures. --- **AI Disclosure** - Model: Claude Fable 5.1 - Platform/Tool: Claude Code - Human Oversight: partially reviewed - Prompt Summary: Add a full-table overwrite mode to the Flink connector, read from a write option and a `write.flink.overwrite-mode` table property, with a guard that blocks a `dynamic` hint when the property is `full`, and tests for the committers and the SQL path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
