stevenzwu commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4364218879
##########
format/spec.md:
##########
@@ -1600,7 +1603,7 @@ The rows in the delete file must be sorted by `file_path`
then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by
one or more column values, and may optionally contain additional columns of the
deleted row.
-Equality delete files must not be added to v4 tables. Equality deletes cannot
be added as an entry to a v4 manifest. Equality delete files referenced by v2
or v3 delete manifests must still be applied in a v4 table.
+Equality delete files must not be added to v4 tables. Equality deletes cannot
be added as an entry to a v4 manifest. Equality delete files referenced by v2
or v3 delete manifests must still be applied in a v4 table. Before column files
are added to a data file, its equality deletes must be written into its
deletion vector. Adding column files raises the data file's data sequence
number, so equality delete files no longer apply to it (see [Column
Files](#column-files)).
Review Comment:
I would suggest move this to the writer requirement subsection in column
files.
not just equality deletes. Due to the v4 colocated DV requirement, v2
position delete files also need to be rewritten to DV. V2 position deletes also
apply data sequence number filter.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
Review Comment:
Maybe simplify it as:
```suggestion
A column file stores the entire value of each field in its `field_ids`. Each
ID must identify a field in the table schema, or a row lineage field required
by [Adding Column Files](#adding-column-files). A schema field must be
top-level or nested in structs, and must not be nested within a list or map. A
field in `field_ids` may also be stored in the data file.
```
A field may be nested in structs, but not within a list or map. For example,
`struct.field` is allowed, but not a field under a struct that is a list
element or a map value.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
Review Comment:
I would move this line/paragraph after line 1685, because line 1685 is
mainly about the column files (data) and later paragraphs are about metadata
(manifests).
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
Review Comment:
Should we describe writer behavior first (before reader)? it would be easier
to understand the reader behavior after writer behavior is understood
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
Review Comment:
nit: `Adding` -> `Writing`, which would also pair well with `Reading`
section title above.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `spec_id` and `partition`:
+ - If an updated field is, or contains, a source field of the partition
spec, `spec_id` and `partition` must be null.
+- `sort_order_id`:
+ - If an updated field is, or contains, a field in the sort order,
`sort_order_id` must be null.
+- `content_stats`:
+ - Stats for each updated field, including its nested fields, must be
computed from the values in the column file that has its ID in `field_ids`, or
omitted. Stats for other fields may be kept.
Review Comment:
I think the wording of this bullet can be tightened up.
```suggestion
- Stats for an updated field and its nested fields must be computed from
the column file that stores it, or omitted. Stats for other fields should be
kept.
```
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `spec_id` and `partition`:
+ - If an updated field is, or contains, a source field of the partition
spec, `spec_id` and `partition` must be null.
+- `sort_order_id`:
+ - If an updated field is, or contains, a field in the sort order,
`sort_order_id` must be null.
+- `content_stats`:
+ - Stats for each updated field, including its nested fields, must be
computed from the values in the column file that has its ID in `field_ids`, or
omitted. Stats for other fields may be kept.
+- `deletion_vector`:
+ - `deletion_vector` must contain all existing deletes of the data file,
including equality deletes.
+- `column_files`:
+ - `column_files` contains the previous entry's column files and the new
column files.
+ - Each updated field ID must be removed from the `field_ids` of the
previous entry's column files, and a column file with no remaining `field_ids`
must be removed.
+ - One of the new column files must have the
`_last_updated_sequence_number` field ID in its `field_ids`. Its value must be
null for each updated row, and for every other row must be the non-null
`_last_updated_sequence_number` that a reader of the previous entry would see.
+
+Deletion vectors in delete manifests that reference the data file must be
removed (see [Deletion Vectors](#deletion-vectors)).
Review Comment:
I don't think we need this sentence here.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
Review Comment:
The intro (line 1661-1671) can be cut down to the scope and the row-count
requirement. The three terms are only used in Adding Column Files, so define
them there.
```
Starting in v4, a data file may store fields in column files listed in
[`column_files`](#column-file-fields). A column file is an Iceberg data file
and must have the same row count as its data file.
```
Regarding the terminology bullets, I still think they are trivial. But it
would be good to get other opinions. They are only used in the `Adding column
files` sub section. If we keep them, they can be moved there.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `spec_id` and `partition`:
+ - If an updated field is, or contains, a source field of the partition
spec, `spec_id` and `partition` must be null.
+- `sort_order_id`:
+ - If an updated field is, or contains, a field in the sort order,
`sort_order_id` must be null.
+- `content_stats`:
+ - Stats for each updated field, including its nested fields, must be
computed from the values in the column file that has its ID in `field_ids`, or
omitted. Stats for other fields may be kept.
+- `deletion_vector`:
+ - `deletion_vector` must contain all existing deletes of the data file,
including equality deletes.
+- `column_files`:
+ - `column_files` contains the previous entry's column files and the new
column files.
+ - Each updated field ID must be removed from the `field_ids` of the
previous entry's column files, and a column file with no remaining `field_ids`
must be removed.
+ - One of the new column files must have the
`_last_updated_sequence_number` field ID in its `field_ids`. Its value must be
null for each updated row, and for every other row must be the non-null
`_last_updated_sequence_number` that a reader of the previous entry would see.
Review Comment:
This can be simplified.
```suggestion
- A new column file must include `_last_updated_sequence_number`, with a
null value for updated rows and the row's current non-null value for other rows.
```
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `spec_id` and `partition`:
+ - If an updated field is, or contains, a source field of the partition
spec, `spec_id` and `partition` must be null.
+- `sort_order_id`:
+ - If an updated field is, or contains, a field in the sort order,
`sort_order_id` must be null.
+- `content_stats`:
+ - Stats for each updated field, including its nested fields, must be
computed from the values in the column file that has its ID in `field_ids`, or
omitted. Stats for other fields may be kept.
+- `deletion_vector`:
Review Comment:
I would drop this bullet and describe writer requirement for equality delete
and v2 position deletes in separate paragraph.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `spec_id` and `partition`:
+ - If an updated field is, or contains, a source field of the partition
spec, `spec_id` and `partition` must be null.
+- `sort_order_id`:
+ - If an updated field is, or contains, a field in the sort order,
`sort_order_id` must be null.
+- `content_stats`:
+ - Stats for each updated field, including its nested fields, must be
computed from the values in the column file that has its ID in `field_ids`, or
omitted. Stats for other fields may be kept.
+- `deletion_vector`:
+ - `deletion_vector` must contain all existing deletes of the data file,
including equality deletes.
+- `column_files`:
+ - `column_files` contains the previous entry's column files and the new
column files.
Review Comment:
I am not we need this bullet. it is also inaccurate. a commit can also
replace/remove column files. I imagine API should allow such update.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `spec_id` and `partition`:
Review Comment:
I don't remember the decision on if updating partition columns is allowed.
If yes, we can have a data file belongs to multiple partitions. This is not
allowed at least in the current v4 tracked file schema which still has the
partition tuple, which is only null if unpartitioned.
If we drop the partition tuple in v4 and replaces it with stats, this can
work. But at least for now, we probably can't do this.
##########
format/spec.md:
##########
@@ -2110,6 +2160,13 @@ Equality deletes are prohibited in v4.
* Upgrading a v2 or v3 table to v4 does not require rewriting data or delete
files
* Readers must continue to apply equality deletes for v2 and v3 tables and for
equality deletes carried over into upgraded v4 tables
+Column updates are added in v4.
+
+* Data files may have column files that store the values of one or more
fields, with one row for each row of the data file (see [Column
Files](#column-files))
+* Adding column files to a data file sets its data sequence number to the
sequence number of the snapshot in which the column files were added
Review Comment:
this doesn't seem completely correct. the data sequence number can be either
null (inheritance in leaf manifest.) or set (root manifest).
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
Review Comment:
> Values in rows deleted by the MODIFIED entry's `deletion_vector` should be
null
maybe simplify as `Values for deleted rows should be null`.
not sure if `should` or `must` is better.
> so a top-level field must be optional in the column file's schema if the
column file stores null for it
Every field in the column file's schema must be optional so that null filler
values can be set for deleted rows
> For a top-level field that is required in the table schema, a non-null
value must be stored in every row that is not deleted by the MODIFIED entry's
`deletion_vector`.
this wording is incorrect. live rows can still have `null` values. we can
just drop this sentence. it also mentioned top-level field.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
Review Comment:
> Row `i` of a column file corresponds to row `i` of the data file.
I found this sentence redundant. The next sentence says the same thing.
> combined by row position
combined -> aligned
> ignore top-level fields
we agreed that it is not limited to top-level anymore. just `ignore fields`
> A null value in a column file does not indicate that a row is deleted.
I think you probably meant the null filler values for deleted rows. but this
sentence doesn't seem necessary and achieve any goal. I would be ok to remove
it.
##########
format/spec.md:
##########
@@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, some fields of a data file can be stored in separate column
files, listed in the data file's [`column_files`](#column-file-fields), in
addition to the data file itself. This section describes column files for data
files.
+
+In this section:
+
+- The **previous entry** is the live entry of the data file before column
files are added.
+- An **updated field** is a field whose ID is in the `field_ids` of an added
column file.
+- An **updated row** is a row whose updated-field values were set by the
update. Other rows keep their current values.
+
+A column file is a valid Iceberg data file: files must use valid Iceberg
formats, schemas, and column projection.
+
+A column file must have the same row count as its data file.
+
+A column file stores the entire value of each field whose ID is in its
`field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the
table schema used to write the column file, or the ID of a row lineage field
required by [Adding Column Files](#adding-column-files). The `field_ids` of a
column file may overlap with the field IDs of the data file. Which file a field
is read from is determined by [Reading Column Files](#reading-column-files).
+
+#### Reading Column Files
+
+A projected field whose top-level field ID is in a column file's `field_ids`,
including `_last_updated_sequence_number`, must be read from that column file.
All other projected fields are read from the data file using [Column
Projection](#column-projection).
+
+Row `i` of a column file corresponds to row `i` of the data file. Column files
and the data file are combined by row position before the deletion vector is
applied. Readers must ignore top-level fields in a column file whose IDs are
not in its `field_ids`. A null value in a column file does not indicate that a
row is deleted.
+
+#### Adding Column Files
+
+When column files are added to a data file, the writer must produce a MODIFIED
entry and a REPLACED entry as described in [Updating Tracking
Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED
entry may be written to different manifests.
+
+A new column file stores the value of each of its listed fields for every row
of the data file, including rows the update did not change. Values in rows
deleted by the MODIFIED entry's `deletion_vector` should be null, so a
top-level field must be optional in the column file's schema if the column file
stores null for it. For a top-level field that is required in the table schema,
a non-null value must be stored in every row that is not deleted by the
MODIFIED entry's `deletion_vector`.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED
entry must match the previous entry, except for the following:
+
+- `spec_id` and `partition`:
+ - If an updated field is, or contains, a source field of the partition
spec, `spec_id` and `partition` must be null.
+- `sort_order_id`:
+ - If an updated field is, or contains, a field in the sort order,
`sort_order_id` must be null.
+- `content_stats`:
+ - Stats for each updated field, including its nested fields, must be
computed from the values in the column file that has its ID in `field_ids`, or
omitted. Stats for other fields may be kept.
+- `deletion_vector`:
+ - `deletion_vector` must contain all existing deletes of the data file,
including equality deletes.
+- `column_files`:
+ - `column_files` contains the previous entry's column files and the new
column files.
+ - Each updated field ID must be removed from the `field_ids` of the
previous entry's column files, and a column file with no remaining `field_ids`
must be removed.
+ - One of the new column files must have the
`_last_updated_sequence_number` field ID in its `field_ids`. Its value must be
null for each updated row, and for every other row must be the non-null
`_last_updated_sequence_number` that a reader of the previous entry would see.
+
+Deletion vectors in delete manifests that reference the data file must be
removed (see [Deletion Vectors](#deletion-vectors)).
+
+Adding column files does not change row IDs (see [Row Lineage](#row-lineage)).
Review Comment:
I think this line can be removed. Row IDs are already unchanged: the
MODIFIED entry keeps the previous entry's `first_row_id`, and a column file
can't contain `_row_id`.
But we should also update line 497, as It points back here for `_row_id`
that also forms a circular reference with the Row Lineage section,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
