anuragmantri commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4520706200
##########
format/spec.md:
##########
@@ -1748,7 +1752,7 @@ The rows in the delete file must be sorted by `file_path`
then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by
one or more column values, and may optionally contain additional columns of the
deleted row.
-Equality delete files must not be added to v4 tables. Equality deletes cannot
be added as an entry to a v4 manifest. Equality delete files referenced by v2
or v3 delete manifests must still be applied in a v4 table.
+Equality delete files must not be added to v4 tables. Equality deletes cannot
be added as an entry to a v4 manifest. Equality delete files referenced by v2
or v3 delete manifests must still be applied in a v4 table. Adding column files
to a data file raises its data sequence number, so equality delete files cannot
be applied to it (see [Writing Column Files](#writing-column-files)).
Review Comment:
Done, thanks.
##########
format/spec.md:
##########
@@ -1801,6 +1804,44 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`, each as a whole,
including all of its nested fields. Each ID must identify a field in the schema
of the file it modifies, which is the table schema for a data file and the
tracked file schema for a leaf manifest, or a row lineage field required by
[Writing Column Files](#writing-column-files). Fields nested within a list or
map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+The rules below apply to column files of data files. Rules for writing column
files of leaf manifests are not defined here.
Review Comment:
Done. I defined base file once in the Column Files intro and used it
everywhere.
##########
format/spec.md:
##########
@@ -157,6 +157,7 @@ Version 4 of the Iceberg spec adds support for relative
locations in metadata, e
* **Delete manifest** -- A file that lists delete files to be associated with
data files at planning time.
* **Data file** -- A file that contains rows of a table.
* **Delete file** -- A file that encodes rows of a table that are deleted by
position or data values.
+* **Column file** -- (v4+) A file that stores the values of one or more fields
of a table or a v4 leaf manifest.
Review Comment:
Done, thanks.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
Review Comment:
Done, thanks.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
Review Comment:
Done, thanks.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
Review Comment:
Done, thanks. If #18526 lands first, I'll add `_last_updated_time` here too.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
Review Comment:
I took the wording but kept "may ... only if". The rule says when a column
file can be used. It doesn't say the update has to fail, which was Gábor's
question earlier.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
Review Comment:
Done, thanks.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
+ - If the added column files change the row,
`_last_updated_sequence_number` must be set to `null` (so that the sequence
number of the snapshot that added the column files replaces the current value).
+ - If the added column files do not change the row and the row is not
deleted, the existing `_last_updated_sequence_number` value, with null resolved
as in [Row lineage assignment](#row-lineage-assignment), must be copied to the
new column file.
+
+#### Reading Column Files
+
+A projected field must be read from the column file that lists the field or a
struct that contains it. This applies to each field nested in a projected
struct. All other projected fields are read from the file it modifies, using
[Column Projection](#column-projection) for a data file. IDs in `field_ids`
that are not projected are ignored.
+
+If a struct is null in a row, each field that the struct contains is null in
that row. This applies when the struct and the nested field are read from
different files.
Review Comment:
Done, thanks.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
+ - If the added column files change the row,
`_last_updated_sequence_number` must be set to `null` (so that the sequence
number of the snapshot that added the column files replaces the current value).
+ - If the added column files do not change the row and the row is not
deleted, the existing `_last_updated_sequence_number` value, with null resolved
as in [Row lineage assignment](#row-lineage-assignment), must be copied to the
new column file.
+
+#### Reading Column Files
+
+A projected field must be read from the column file that lists the field or a
struct that contains it. This applies to each field nested in a projected
struct. All other projected fields are read from the file it modifies, using
[Column Projection](#column-projection) for a data file. IDs in `field_ids`
that are not projected are ignored.
Review Comment:
Done. I kept "for a data file" after Column Projection, since that section
is about data files.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
Review Comment:
Done, with one change: the ID of a field that contains the struct also
counts. Otherwise a column file that rewrites all of `customer` would break the
rule when `customer.address` becomes null.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
Review Comment:
Done. I kept a short clause on why, since Russell asked for the reason
earlier.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
+ - If the added column files change the row,
`_last_updated_sequence_number` must be set to `null` (so that the sequence
number of the snapshot that added the column files replaces the current value).
+ - If the added column files do not change the row and the row is not
deleted, the existing `_last_updated_sequence_number` value, with null resolved
as in [Row lineage assignment](#row-lineage-assignment), must be copied to the
new column file.
+
+#### Reading Column Files
+
+A projected field must be read from the column file that lists the field or a
struct that contains it. This applies to each field nested in a projected
struct. All other projected fields are read from the file it modifies, using
[Column Projection](#column-projection) for a data file. IDs in `field_ids`
that are not projected are ignored.
+
+If a struct is null in a row, each field that the struct contains is null in
that row. This applies when the struct and the nested field are read from
different files.
+
+If a column file does not store a field that must be read from it, the reader
must use the field's `initial-default`, or null if no default is set.
Review Comment:
Done, but I used "the schema of its base file" instead of the table schema.
That also covers a leaf manifest column file that lists `content_stats`, which
gains a stats field when the table gets a new column.
##########
format/spec.md:
##########
@@ -999,6 +1002,7 @@ The contents of a file are part of a snapshot if its
tracking `status` is **live
- `column_files`:
- `column_files` may only be set if the tracked file is a data file or
a v4 leaf manifest.
- A field ID may appear in `field_ids` of at most one column file in a
tracked file's `column_files`.
+ - If a column file lists a struct, no other column file in the tracked
file's `column_files` may list a field nested in that struct.
Review Comment:
Done, thanks.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
Review Comment:
Done, using base file.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
Review Comment:
Agreed, I dropped both v4 mentions from the intro. The rule that only v4
leaf manifests can have column files is still in the tracked file requirements.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
Review Comment:
Done. A leaf manifest's sequence numbers stay non-null because the root
manifest requires them for every tracked file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
