gaborkaszab commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4422362593
########## format/spec.md: ########## @@ -1801,6 +1804,40 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file may store fields in column files listed in [`column_files`](#column-file-fields). A column file is an Iceberg data file and must have the same row count as its data file. + +A column file stores the entire value of each field in its `field_ids`. Each ID must identify a field in the table schema, or a row lineage field required by [Writing Column Files](#writing-column-files). A schema field must be top-level or nested in structs, must not be nested within a list or map, and must not be, or contain, a source field of the data file's partition spec. + +#### Writing Column Files + +In this subsection, the **previous entry** is the live entry of the data file before column files are added, an **updated field** is a table schema field whose ID is in the `field_ids` of an added column file, and an **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values for deleted rows should be null, so every field in `field_ids` must be optional in the column file's schema. + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +Adding column files raises the data file's data sequence number, so deletes in delete manifests would stop applying to it. The MODIFIED entry's `deletion_vector` must therefore contain all deletes that apply to the data file, from equality delete files, v2 position delete files and deletion vectors in delete manifests. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for `deletion_vector` and the following: + +- `sort_order_id`: + - If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for an updated field and its nested fields must be computed from the column file that lists it, or omitted. +- `tight_stats`: + - `tight_stats` must be false or omitted. +- `column_files`: + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One new column file must list `_last_updated_sequence_number` in its `field_ids`, with a null value for updated rows and the row's current non-null value for other rows that are not deleted. + +#### Reading Column Files + +A projected field whose ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). Review Comment: I think you're right, we probably don't want to call out these "special" fields on the read side. We can say that if the projected field ID is covered by one of the column files then we read that field from the respective column file. If it's not covered, then read from the base file. On the write side though, I think we have to spell out that when writing column files, at least one of them should have the `_LUSN` (and later on the timestamp one too) in the file and also in `field_ids` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
