pvary commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4324916508
##########
format/spec.md:
##########
@@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Updates
+
+Starting in v4, the values of one or more fields in a data file can be updated
without rewriting the data file by adding column files using the
[`column_files`](#column-file-fields) field. This section describes column
files for data files.
+
+In this section, the **previous entry** is the live entry of the data file
before a column update, the **MODIFIED entry** is the entry produced by the
column update, an **updated field** is a field whose ID is in the `field_ids`
of a column file added by the column update, and an **updated row** is a row
modified by the column update.
+
+#### Column Files
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files).
+
+A column file must have the same row count as its data file, and row `i` of
the column file stores values for row `i` of the data file. Each row stores the
value of each listed field after the column update, except that values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage
fields are written as described in [Adding Column Files](#adding-column-files).
+
+Column file requirements:
+
+- Each top-level field must be optional in the column file's schema, even if
the field is required in the table schema.
+- For a top-level field that is required in the table schema, a non-null value
must be stored in every row that is not deleted by the MODIFIED entry's
`deletion_vector`.
+- A column file must not contain the [reserved](#reserved-field-ids) `_row_id`
or `_pos` fields.
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Readers must ignore top-level fields in a column file whose IDs are not in its
`field_ids`. A null value in a column file does not indicate that a row is
deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+The MODIFIED entry must have the previous metadata of the data file, except
for the following:
+
+- `tracking`:
+ - `tracking.status` is MODIFIED.
+ - `tracking.snapshot_id` must not be null.
+ - `tracking.modified_snapshot_id` must record the snapshot ID in which the
column files were added.
+ - `tracking.sequence_number` is the sequence number of the snapshot in
which the column files were added. See [Sequence Number
Inheritance](#sequence-number-inheritance).
+- `spec_id` and `partition`:
+ - If an updated field is, or contains, the source of a field in the
previous entry's partition spec, `partition` must be the partition tuple shared
by the rows that are not deleted, or `spec_id` and `partition` must be null.
+ - If `spec_id` and `partition` are set to null, each source field of an
identity partition field in the previous entry's partition spec that is missing
from the data file must be stored in a column file of the MODIFIED entry that
has the source's top-level field ID in its `field_ids`.
+- `sort_order_id`:
+ - If an updated field is, or contains, the source of a field in the
previous entry's sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for each updated field, including its nested fields, must be
computed from the values in the column file that has its ID in `field_ids`, or
omitted. Stats for other fields may be kept.
+- `deletion_vector`:
+ - `deletion_vector` must contain all deletes from the previous entry's
`deletion_vector` and from delete files and deletion vectors in delete
manifests that apply to the previous entry, including equality delete files.
+- `column_files`:
+ - New column files are added to the previous entry's `column_files`.
+ - Each updated field ID must be removed from the `field_ids` of the
previous entry's column files, and a column file with no remaining `field_ids`
must be removed.
+ - One of the new column files must have the
`_last_updated_sequence_number` field ID in its `field_ids`. Its value must be
null for each updated row, and must be the row's current
`_last_updated_sequence_number`, read from the previous entry, for every other
row that is not deleted.
Review Comment:
```suggestion
- One of the new column files must have the
`_last_updated_sequence_number` field ID in its `field_ids`. Its value must be
null for each updated row, and must be the row's current
`_last_updated_sequence_number`, the non-null value a reader of the previous
entry would see, for every other row that is not deleted.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
