RussellSpitzer commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4498397754
########## format/spec.md: ########## @@ -1801,6 +1804,43 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file or v4 leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). A column file can use any supported Iceberg file format and must have rows that align with the file it modifies. + +A column file stores the fields listed in its `field_ids`, each as a whole, including all of its nested fields. Each ID must identify a field in the schema of the file it modifies, which is the table schema for a data file and the tracked file schema for a leaf manifest, or a row lineage field required by [Writing Column Files](#writing-column-files). Fields nested within a list or map must not be listed in `field_ids`. Review Comment: Suggested text changes - **In Column Files** A column file stores the fields listed in its `field_ids`. A listed struct includes all fields nested in it. In this section, a struct _contains_ a field if the field is nested in the struct at any depth. Each ID in `field_ids` must identify a field in the schema of the file it modifies, or a row lineage field required by [Writing Column Files](#writing-column-files). The schema of a data file is the table schema. The schema of a leaf manifest is the tracked file schema. Fields nested in a list or map must not be listed in `field_ids`. If a column file of an entry lists a struct, no other column file of that entry can list a field that the struct contains. **In Writing Column Files** The added column files must list each field that the update changes, or a struct that contains that field. If an update changes whether a struct is null, the update changes that struct. Changing the ColumnFile Bullet - - Each field listed in an added column file must be removed from the `field_ids` of the previous entry's column files. Each field that a listed struct contains must also be removed. A column file with no remaining `field_ids` must be removed. **In Reading Column Files** A projected field must be read from the column file that lists the field or a struct that contains it. All other projected fields are read from the file it modifies, using [Column Projection](#column-projection) for a data file. If a struct is null in a row, each field that the struct contains is null in that row. This applies when the struct and the nested field are read from different files. If a column file does not store a field that must be read from it, the reader must use the field's `initial-default`, or null if no default is set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
