stevenzwu commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4342389812
########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). Review Comment: > , and row `i` of the column file stores values for row `i` of the data file we probably don't need this part. `same row count` is sufficient. we can talk more about position alignment when talking about read stitching. > after the column update don't need to say column update > , except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null I would drop `except that`. also maybe we should just drop it and cover it in the writer behavior section. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. Review Comment: I am not sure we need this terminology paragraph. e.g. modified entry, updated field, updated row seem trivial. if we do, a bullet list would be readable. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. Review Comment: Anoop's tight_stats spec PR #18585 should be sufficient for column files too. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. Review Comment: we probably just need to talk about how writers handle DV marked rows, i.e. null filler value. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. Review Comment: good question. Stats aggregation for leaf manifest files likely happens in iceberg-core commit code path. if the modified entry into a new manifest file, the aggregation would happen naturally. If the leaf manifest file itself is updated via column overlay, the commit code should re-aggregate them. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. Review Comment: This paragraph isn't quite clear. I would say something along this line: ```suggestion Starting in v4, some columns can be stored in separate column files in addition to the base data file. ``` ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates Review Comment: I am not sure we need a separate section for column updates. column files can be used for adding or updating columns. we may fold content directly into the "Column File" section below. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. Review Comment: +1 on dropping this. I thought `_pos` was only populated by reader and is never persisted. We didn't say it for data files, and we don't need to say it for column files. I have heard use cases where we can store `_row_id` in column files to perform row id reassignment to fix up corrupted row ids. We probably don't need to restrict it in the spec. "Adding column files does not change row IDs" is the part that matters. But maybe we can cover it in the writer rules. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. Review Comment: I think the reason is allow null filler values for DV marked rows as the 2nd bullet. maybe the two can merge to provide the context why we are modifying the column file schema to optional. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. Review Comment: +1. But we can't do it now. Hopefully, my row timestamp spec PR #18526 can be merged before this :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
