mikebewell commented on PR #18796: URL: https://github.com/apache/iceberg/pull/18796#issuecomment-6643392369
Manual test on real tables, Flink 2.3.0 local cluster, REST catalog, connector built from this PR's commit. Two tables partitioned by `days(ts)`, each holding one file per day for 10-26, 10-27 and 10-28 before every statement. T1 has no table property; T2 has `write.flink.overwrite-mode=full`. Each `INSERT OVERWRITE` source has rows for 10-27 and 10-28 only, so a dynamic overwrite keeps the 10-26 row and a full overwrite removes it. Results read from `$snapshots` and a `SELECT`. | # | Statement | Snapshot summary and rows | |---|---|---| | 1 | T1, `INSERT OVERWRITE`, no hint | `overwrite`, `replace-partitions=true`, added 2, deleted 2, total 3; 10-26 row stays | | 2 | T1, hint `overwrite-mode=full` | `overwrite`, no `replace-partitions`, added 2, deleted 3, total 2; 10-26 row gone | | 3 | T1, hint `full`, source with 0 rows | no new snapshot, rows unchanged; log `Skip full table overwrite for checkpoint ... due to no data files.` | | 4 | T2, `INSERT INTO` | `append`, added 1, total 3 | | 5 | T2, `INSERT OVERWRITE`, no hint | `overwrite`, no `replace-partitions`, added 2, deleted 3, total 2; 10-26 row gone | | 6 | T2, hint `overwrite-mode=dynamic` | rejected: `Invalid overwrite-mode=dynamic: table property write.flink.overwrite-mode=full requires a full overwrite`; no new snapshot | Not covered manually, covered by unit tests: delete files, files under an older partition spec, the `PARTITION (...)` rejection (these tables have no identity partition column), the `IcebergSink` v2 path (the SQL sink used `IcebergFilesCommitter`). The dynamic sink was not tested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
