RussellSpitzer commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4418799384
########## format/spec.md: ########## @@ -1801,6 +1804,40 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file may store fields in column files listed in [`column_files`](#column-file-fields). A column file is an Iceberg data file and must have the same row count as its data file. + +A column file stores the entire value of each field in its `field_ids`. Each ID must identify a field in the table schema, or a row lineage field required by [Writing Column Files](#writing-column-files). A schema field must be top-level or nested in structs, must not be nested within a list or map, and must not be, or contain, a source field of the data file's partition spec. + +#### Writing Column Files + +In this subsection, the **previous entry** is the live entry of the data file before column files are added, an **updated field** is a table schema field whose ID is in the `field_ids` of an added column file, and an **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values for deleted rows should be null, so every field in `field_ids` must be optional in the column file's schema. + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +Adding column files raises the data file's data sequence number, so deletes in delete manifests would stop applying to it. The MODIFIED entry's `deletion_vector` must therefore contain all deletes that apply to the data file, from equality delete files, v2 position delete files and deletion vectors in delete manifests. Review Comment: You could add a foot note explaining why but i'm not sure we need to do that in the spec -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
