pvary commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4324895155
##########
format/spec.md:
##########
@@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Updates
+
+Starting in v4, the values of one or more fields in a data file can be updated
without rewriting the data file by adding column files using the
[`column_files`](#column-file-fields) field. This section describes column
files for data files.
+
+In this section, the **previous entry** is the live entry of the data file
before a column update, the **MODIFIED entry** is the entry produced by the
column update, an **updated field** is a field whose ID is in the `field_ids`
of a column file added by the column update, and an **updated row** is a row
modified by the column update.
+
+#### Column Files
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files).
+
+A column file must have the same row count as its data file, and row `i` of
the column file stores values for row `i` of the data file. Each row stores the
value of each listed field after the column update, except that values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage
fields are written as described in [Adding Column Files](#adding-column-files).
+
+Column file requirements:
+
+- Each top-level field must be optional in the column file's schema, even if
the field is required in the table schema.
+- For a top-level field that is required in the table schema, a non-null value
must be stored in every row that is not deleted by the MODIFIED entry's
`deletion_vector`.
+- A column file must not contain the [reserved](#reserved-field-ids) `_row_id`
or `_pos` fields.
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Readers must ignore top-level fields in a column file whose IDs are not in its
`field_ids`. A null value in a column file does not indicate that a row is
deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+The MODIFIED entry must have the previous metadata of the data file, except
for the following:
+
+- `tracking`:
+ - `tracking.status` is MODIFIED.
+ - `tracking.snapshot_id` must not be null.
Review Comment:
```suggestion
- `tracking.snapshot_id` must be the previous entry's (inherited)
snapshot ID.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
