RussellSpitzer commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4446587346
##########
format/spec.md:
##########
@@ -1801,6 +1804,40 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file may store fields in column files listed in
[`column_files`](#column-file-fields). A column file can use any supported
Iceberg file format and must have rows that align with the data file it
modifies.
+
+A column file is responsible for all the values of the fields listed in its
`field_ids`. Each ID must identify a field in the table schema, or a row
lineage field required by [Writing Column Files](#writing-column-files). A
schema field must be top-level or nested in structs, and must not be nested
within a list or map.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the data file. Values that did not change are still included, and values for
deleted rows should be null.
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `partition`:
+ - If an added column file lists a field that is, or contains, a source
field of the data file's partition spec, every row that is not deleted must
have the same partition values after the update, and `partition` must be set to
those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+- `tight_stats`:
+ - `tight_stats` must be false or omitted.
+- `deletion_vector`:
+ - Every delete that applied to the data file before the column files were
added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field ID listed in an added column file must be removed from the
`field_ids` of the previous entry's column files, and a column file with no
remaining `field_ids` must be removed.
+ - One new column file must list `_last_updated_sequence_number` in its
`field_ids`, with a null value for rows the update changed and the row's
current non-null value for other rows that are not deleted.
Review Comment:
Yes I think we are writing this requirement too passively
```
- `column_files`:
- Each field ID listed in an added column file must be removed from the
`field_ids` of the previous entry's column files, and a column file with no
remaining `field_ids` must be removed.
- One of the added column files must take responsibility for
`_last_updated_sequence_number` by listing it in its `field_ids`, replacing the
data file or previous column file that stored it. In that column file:
- Rows the update changed must store null, so they inherit the
MODIFIED entry's data sequence number.
- Other rows that are not deleted must store the non-null
`_last_updated_sequence_number` read from the previous entry.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
