stevenzwu commented on code in PR #18526: URL: https://github.com/apache/iceberg/pull/18526#discussion_r4288883521
########## format/spec.md: ########## @@ -2110,6 +2135,18 @@ Equality deletes are prohibited in v4. * Upgrading a v2 or v3 table to v4 does not require rewriting data or delete files * Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables +Row lineage changes: + +* When writing a manifest, ADDED entries and entries MODIFIED by column files must be written with a null `snapshot_timestamp` +* When an ADDED or MODIFIED entry has a null `snapshot_timestamp`, readers must assign it the snapshot's `timestamp-ms` converted to microseconds in the root manifest, or the leaf manifest's `snapshot_timestamp` in a leaf manifest Review Comment: > if you rewrite a data file that was added prior to V4 it ends up with a null timestamp. this would generates a new entry in pre-v4 manifest file that has a different manifest schema and file format and doesn't have the `snapshot_timestamp` field. > If you then rewrite a second time, I think you end up populating the the timestamp field with the second rewrite because you can no longer determine the change? for v4 tables, the writer is required to persist the carried-over `_last_update_time` values for non-modified rows and `null` for modified rows. But in this case, both non-modified and modified rows would have `null` values. Inheritance kick in for both categories, which would produce wrong timestamp values for non-modified rows. Please let me know if I understand it correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
