singhpk234 commented on code in PR #17783: URL: https://github.com/apache/iceberg/pull/17783#discussion_r4056239013
########## format/spec.md: ########## @@ -1416,6 +1417,8 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row. +Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry. Review Comment: > Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry why can't one convert eq to dv and write that instead ? i think we are overspecifying here ########## format/spec.md: ########## @@ -1352,13 +1353,13 @@ Notes: ### Delete Formats -This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. +This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. Equality delete files must not be added to v4 tables, but existing equality delete files are valid. There are different formats for encoding row-level deletes: * Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap * Position delete files identify deleted rows by file location and row position (**deprecated** in v3) -* Equality delete files identify deleted rows by the value of one or more columns +* Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4) Review Comment: whats the difference between prohibited vs deprecated in this context, why not say deprecated in v4 ? are we saying its recommendation or disallowing it if disallowing then deprecated seems more sensible. ########## format/spec.md: ########## @@ -62,6 +62,7 @@ The full set of changes are listed in [Appendix E](#version-3). Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities: * Support for [relative locations](#file-locations-in-metadata) in metadata fields +* Writing new [equality deletes](#equality-delete-files) is no longer allowed Review Comment: whats the expectation for the writer ? whats the expectation for the catalog for example rest ? ... do we need to elaborate that ? ########## format/spec.md: ########## @@ -1913,6 +1916,12 @@ Reading v4 metadata: * Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution)) * When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification)) +Row-level delete changes: + +* Equality delete files must not be added to v4 tables and equality deletes cannot be added as an entry to a v4 manifest +* Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables +* Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files Review Comment: is this required ? this is true irrespective of whether table has row level deletes or not ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
