anuragmantri commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4386267387
########## format/spec.md: ########## Review Comment: Update: the rule now lives in Writing Column Files and covers v2 position delete files and deletion vectors in delete manifests too. The sentence in Equality Delete Files is a pointer. ########## format/spec.md: ########## @@ -553,7 +554,7 @@ Note that: ### Partitioning Review Comment: Update: superseded. Column files can no longer list partition source fields, so the partition tuple never changes. See my update on the other partition thread. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. Review Comment: Update: after Steven's comment, column files can no longer list a partition source field. The partition tuple is still in the tracked file and null means unpartitioned, so nulling it doesn't fit. That also settles the Hive question above. We can revisit once the tuple is dropped. ########## format/spec.md: ########## @@ -1600,7 +1603,7 @@ The rows in the delete file must be sorted by `file_path` then `pos` to optimize Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row. -Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. +Equality delete files must not be added to v4 tables. Equality deletes cannot be added as an entry to a v4 manifest. Equality delete files referenced by v2 or v3 delete manifests must still be applied in a v4 table. Before column files are added to a data file, its equality deletes must be written into its deletion vector. Adding column files raises the data file's data sequence number, so equality delete files no longer apply to it (see [Column Files](#column-files)). Review Comment: Agreed, done. I left a one-line pointer in Equality Delete Files because Gábor asked for one there. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. Review Comment: Agreed, done. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. Review Comment: Agreed, done. I moved the terms to the start of Writing Column Files and kept them for now. Happy to drop them if others agree. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `spec_id` and `partition`: + - If an updated field is, or contains, a source field of the partition spec, `spec_id` and `partition` must be null. +- `sort_order_id`: + - If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. Review Comment: Agreed, done. I used "lists" instead of "stores", because an older column file can still hold a field it no longer lists. I also dropped the second sentence, since the lead-in already says the other fields must match the previous entry. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. Review Comment: Agreed, done. I scoped "must be optional" to the fields in `field_ids`, because map keys can't be optional and only listed fields hold the null fillers. I kept "should" for the nulls so a deletion vector committed while the file is being written doesn't invalidate it. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `spec_id` and `partition`: + - If an updated field is, or contains, a source field of the partition spec, `spec_id` and `partition` must be null. +- `sort_order_id`: + - If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: Review Comment: Agreed, done. ########## format/spec.md: ########## @@ -553,7 +554,7 @@ Note that: ### Partitioning -Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. +Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. For a data file with column files, this requirement applies only to rows that are not deleted, using values read with its column files. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. Review Comment: Update: the invariant now holds because column files can't list partition source fields, so `spec_id` and `partition` stay as they are. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `spec_id` and `partition`: + - If an updated field is, or contains, a source field of the partition spec, `spec_id` and `partition` must be null. +- `sort_order_id`: + - If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all existing deletes of the data file, including equality deletes. +- `column_files`: + - `column_files` contains the previous entry's column files and the new column files. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and for every other row must be the non-null `_last_updated_sequence_number` that a reader of the previous entry would see. + +Deletion vectors in delete manifests that reference the data file must be removed (see [Deletion Vectors](#deletion-vectors)). + +Adding column files does not change row IDs (see [Row Lineage](#row-lineage)). Review Comment: Agreed, removed. The Row Lineage pointer now covers `_last_updated_sequence_number` only. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `spec_id` and `partition`: + - If an updated field is, or contains, a source field of the partition spec, `spec_id` and `partition` must be null. +- `sort_order_id`: + - If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all existing deletes of the data file, including equality deletes. +- `column_files`: + - `column_files` contains the previous entry's column files and the new column files. Review Comment: Agreed, done. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `spec_id` and `partition`: + - If an updated field is, or contains, a source field of the partition spec, `spec_id` and `partition` must be null. +- `sort_order_id`: + - If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all existing deletes of the data file, including equality deletes. +- `column_files`: + - `column_files` contains the previous entry's column files and the new column files. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and for every other row must be the non-null `_last_updated_sequence_number` that a reader of the previous entry would see. + +Deletion vectors in delete manifests that reference the data file must be removed (see [Deletion Vectors](#deletion-vectors)). Review Comment: Agreed, removed. What supersedes a pre-v4 delete manifest DV when a v4 DV is written isn't specified for DV-only changes either, so that's probably a #16025 follow-up. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). Review Comment: Note: "Adding Column Files" is now "Writing Column Files" and comes before "Reading Column Files", per Steven's comments. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). Review Comment: Agreed, done. I dropped the last sentence, since Reading Column Files already says which file a field is read from. I'm still thinking about how nested fields are read and will update that language soon. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `spec_id` and `partition`: Review Comment: I noted from one of the old sync that it is allowed on the assumption that the partition tuple would be dropped. Since it's still there, I agree we can't do this now. Column files must not list a partition source field. We can revisit once the tuple goes. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files Review Comment: Agreed, done. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. Review Comment: Agreed, done. I dropped the "ignore fields" sentence entirely, since the rule above already reads any field that isn't listed in a column file from the data file. ########## format/spec.md: ########## @@ -2110,6 +2160,13 @@ Equality deletes are prohibited in v4. * Upgrading a v2 or v3 table to v4 does not require rewriting data or delete files * Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables +Column updates are added in v4. + +* Data files may have column files that store the values of one or more fields, with one row for each row of the data file (see [Column Files](#column-files)) +* Adding column files to a data file sets its data sequence number to the sequence number of the snapshot in which the column files were added Review Comment: Agreed, done. It now says the data sequence number is raised to the snapshot's, so it holds whether the entry stores it or inherits it. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files Review Comment: Agreed, done. ########## format/spec.md: ########## @@ -1653,6 +1656,53 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, some fields of a data file can be stored in separate column files, listed in the data file's [`column_files`](#column-file-fields), in addition to the data file itself. This section describes column files for data files. + +In this section: + +- The **previous entry** is the live entry of the data file before column files are added. +- An **updated field** is a field whose ID is in the `field_ids` of an added column file. +- An **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file must have the same row count as its data file. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). The `field_ids` of a column file may overlap with the field IDs of the data file. Which file a field is read from is determined by [Reading Column Files](#reading-column-files). + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Row `i` of a column file corresponds to row `i` of the data file. Column files and the data file are combined by row position before the deletion vector is applied. Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values in rows deleted by the MODIFIED entry's `deletion_vector` should be null, so a top-level field must be optional in the column file's schema if the column file stores null for it. For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `spec_id` and `partition`: + - If an updated field is, or contains, a source field of the partition spec, `spec_id` and `partition` must be null. +- `sort_order_id`: + - If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all existing deletes of the data file, including equality deletes. +- `column_files`: + - `column_files` contains the previous entry's column files and the new column files. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and for every other row must be the non-null `_last_updated_sequence_number` that a reader of the previous entry would see. Review Comment: Agreed, done, with three changes: "One new column file", because a field ID can be listed in only one column file; "list in its `field_ids`" instead of "include", because readers only use listed fields; and "that are not deleted", so it doesn't conflict with the null fillers for deleted rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
