gaborkaszab commented on code in PR #16025:
URL: https://github.com/apache/iceberg/pull/16025#discussion_r3722214174
##########
format/spec.md:
##########
@@ -691,21 +707,118 @@ The `manifest_entry` struct consists of the following
fields:
| | _optional_ | **`4 file_sequence_number`** | `long`
| File sequence number indicating
when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file`
`struct` (see below) | File path, partition tuple,
metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
+ The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
-When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
+ When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
-When a file is replaced or deleted from the dataset, its manifest entry fields
store the snapshot ID in which the file was deleted and status 2 (deleted). The
file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+ When a file is replaced or deleted from the dataset, its manifest entry
fields store the snapshot ID in which the file was deleted and status 2
(deleted). The file may be deleted from the file system when the snapshot in
which it was deleted is garbage collected, assuming that older snapshots have
also been garbage collected [1].
-Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
-The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
+ The `sequence_number` field represents the data sequence number and must
never change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status
is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry
must include both sequence numbers explicitly.
-Notes:
+ Notes:
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
+ 1. Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected. But this is harder to
detect and requires finding the diff of multiple snapshots. It is easier to
track what files are deleted in a snapshot and delete them when that snapshot
expires. It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number`
and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3:
DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content
stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
*required* | Writer format version. V4 writers must produce `format_version` 4.
|
+ | 100 | **`location`** | `string` | *required* | *required* | Location of
the file or manifest. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* |
Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the
partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID
representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of
records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* |
*optional* | Column stats. See [Column Stats
Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* |
*optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional*
| Split offsets for the data or equality delete file. Must be sorted ascending.
|
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* |
Field ids for row equality in equality delete files. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
*optional* | Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
*optional* | Column update files associated with this entry. |
Review Comment:
"Column update files associated with this entry"
I wouldn't mention update, simple "Column files". In the future we might
want to add column families, where column files aren't the result of an update.
I'd leave this flexibility with the comment not breaking later on.
##########
format/spec.md:
##########
@@ -691,21 +707,118 @@ The `manifest_entry` struct consists of the following
fields:
| | _optional_ | **`4 file_sequence_number`** | `long`
| File sequence number indicating
when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file`
`struct` (see below) | File path, partition tuple,
metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
+ The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
-When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
+ When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
-When a file is replaced or deleted from the dataset, its manifest entry fields
store the snapshot ID in which the file was deleted and status 2 (deleted). The
file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+ When a file is replaced or deleted from the dataset, its manifest entry
fields store the snapshot ID in which the file was deleted and status 2
(deleted). The file may be deleted from the file system when the snapshot in
which it was deleted is garbage collected, assuming that older snapshots have
also been garbage collected [1].
-Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
-The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
+ The `sequence_number` field represents the data sequence number and must
never change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status
is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry
must include both sequence numbers explicitly.
-Notes:
+ Notes:
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
+ 1. Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected. But this is harder to
detect and requires finding the diff of multiple snapshots. It is easier to
track what files are deleted in a snapshot and delete them when that snapshot
expires. It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number`
and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3:
DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content
stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
*required* | Writer format version. V4 writers must produce `format_version` 4.
|
+ | 100 | **`location`** | `string` | *required* | *required* | Location of
the file or manifest. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* |
Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the
partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID
representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of
records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* |
*optional* | Column stats. See [Column Stats
Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* |
*optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional*
| Split offsets for the data or equality delete file. Must be sorted ascending.
|
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* |
Field ids for row equality in equality delete files. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
*optional* | Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
*optional* | Column update files associated with this entry. |
+
+ Value 1 (POSITION_DELETES) is not used in v4. Writers must not produce
`content_type` 1.
+
+ V4 leaf data manifests must only contain entries with `content_type` 0
(DATA); V4 leaf delete manifests must only contain entries with `content_type`
2 (EQUALITY DELETES). A root manifest may reference V1-V3 manifests; V1-V3 leaf
manifest references must have `format_version` set to 0.
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null
otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null
otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null
otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is 3 or 4 (manifests).
+ - `split_offsets` must be null when `content_type` is 3 or 4 (manifests).
+ - `tracking.deleted_positions` and `tracking.replaced_positions` must be
null when `content_type` is not 3 or 4.
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | *required* | Used to track additions,
deletions, replacements, and modifications. When a data file's
`deletion_vector` or `column_files` change, REPLACED marks the prior version of
the entry and MODIFIED marks the new, live version. For leaf manifest entries,
MODIFIED marks a live manifest whose `dv` changed. Deletes are not used in
scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID
where the file was added or deleted. Inherited when null. Optional for leaf
manifests, required for root. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | *optional* | Snapshot
ID where the deletion vector was added. Inherited when null. Must be null when
`deletion_vector` is null. |
+ | 3 | **`sequence_number`** | `long` | *optional* | *optional* | Data
sequence number of the file. Inherited when null and status is 1 (ADDED). Must
equal `file_sequence_number` if `content_type` is 3 or 4. Optional for leaf
manifests, required for root. |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | *optional* | File
sequence number indicating when the file was added. Inherited when null and
status is ADDED. Must equal `sequence_number` if `content_type` is 3 or 4. |
+ | 142 | **`first_row_id`** | `long` | *optional* | *optional* | The
`_row_id` for the first row in the data file if `content_type` is 0. If
`content_type` is 3, this is the starting `_row_id` to assign to rows added by
ADDED data files. See [First Row ID Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were deleted in this snapshot. |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were replaced in this snapshot. |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 155 | **`location`** | `string` | *required* | *required* | Location of
the Puffin file. |
+ | 144 | **`offset`** | `long` | *required* | *required* | Offset in the
file where the content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | *required* | Length of
the referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | *required* | Cardinality
of the deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 504 | **`added_files_count`** | `long` | *required* | *required* | Count
of entries with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `long` | *required* | *required* |
Count of entries with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `long` | *required* | *required* |
Count of entries with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `long` | *required* | *required* |
Count of entries with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `long` | *required* | *required* |
Count of entries with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | *required* | Total
number of rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | *required* |
Total number of rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | *required* |
Total number of rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | *required* |
Total number of rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | *required* |
Total number of rows in MODIFIED entries. |
+ | 516 | **`min_sequence_number`** | `long` | *required* | *required* |
Minimum data sequence number of all live entries in the manifest. |
+ | 522 | **`dv`** | `binary` | *optional* | *optional* | Roaring bitmap of
entry positions in the manifest that are not live in the current snapshot. |
+ | 523 | **`dv_cardinality`** | `long` | *optional* | *optional* |
Cardinality of the manifest deletion vector. Must be set when `dv` is non-null;
must be null otherwise. |
+
+ **`column_file` struct (element 159 of `column_files`, field 158)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 161 | **`format_version`** | `int` | *required* | *required* | Format
version of this column file. |
Review Comment:
nit: "this column file" to me this sounds right in the Java class, but feels
slightly odd in the spec. Maybe "the column file" instead of this? Other
occasions below
##########
format/spec.md:
##########
@@ -685,21 +701,100 @@ The `manifest_entry` struct consists of the following
fields:
| | _optional_ | **`4 file_sequence_number`** | `long`
| File sequence number indicating
when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file`
`struct` (see below) | File path, partition tuple,
metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
+ The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
-When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
+ When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
-When a file is replaced or deleted from the dataset, its manifest entry fields
store the snapshot ID in which the file was deleted and status 2 (deleted). The
file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+ When a file is replaced or deleted from the dataset, its manifest entry
fields store the snapshot ID in which the file was deleted and status 2
(deleted). The file may be deleted from the file system when the snapshot in
which it was deleted is garbage collected, assuming that older snapshots have
also been garbage collected [1].
-Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
-The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
+ The `sequence_number` field represents the data sequence number and must
never change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status
is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry
must include both sequence numbers explicitly.
-Notes:
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected. But this is harder to
detect and requires finding the diff of multiple snapshots. It is easier to
track what files are deleted in a snapshot and delete them when that snapshot
expires. It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number`
and `snapshot_id` to inherit are always available.
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3:
DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content
stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) |
*required* | *required* | Writer format version. V4 writers must produce
`writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of
the file or manifest. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* |
Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the
partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID
representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of
records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* |
*optional* | Column stats. See [Column Stats
Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* |
*optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional*
| Split offsets for the data file. Must be sorted ascending. |
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* |
Field ids for row equality in equality delete files. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
*optional* | Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional*
| Column update files associated with this entry. |
+
+ Value 1 (POSITION_DELETES) is not used in v4. Writers must not produce
`content_type` 1.
+
+ V4 leaf data manifests must only contain entries with `content_type` 0
(DATA); V4 leaf delete manifests must only contain entries with `content_type`
2 (EQUALITY DELETES). A root manifest may reference V1-V3 manifests; V1-V3 leaf
manifest references must have `writer_format_version` set to 0.
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null
otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null
otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null
otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is not 0.
+ - `tracking.deleted_positions` and `tracking.replaced_positions` must be
null when `content_type` is not 3 or 4.
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED) | *required* | *required* | Used to track additions, deletions, and
replacements. REPLACED indicates entries with data column updates or
`deletion_vector` changes. Deleted entries are required when the snapshot has a
non-null parent. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID
where the file was added or deleted. Inherited when null. Optional for leaf
manifests, required for root. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | *optional* | Snapshot
ID where the deletion vector was added. Inherited when null. Must be null when
`deletion_vector` is null. |
Review Comment:
If I don't miss anything, then the latest changes don't add the column file
related snapshot ID to tracking. I have that in [the implementation
PR](https://github.com/apache/iceberg/pull/16285/changes#diff-e250410e39909d733ebfbcec883d6e7dc8f3cba3884730dbfbaca19ac3e12e11R68)
with field ID 160. I used the name `latest_column_file_snapshot_id` that might
be too long, but describes exactly what it is.
##########
format/spec.md:
##########
@@ -691,21 +707,118 @@ The `manifest_entry` struct consists of the following
fields:
| | _optional_ | **`4 file_sequence_number`** | `long`
| File sequence number indicating
when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file`
`struct` (see below) | File path, partition tuple,
metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
+ The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
-When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
+ When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
-When a file is replaced or deleted from the dataset, its manifest entry fields
store the snapshot ID in which the file was deleted and status 2 (deleted). The
file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+ When a file is replaced or deleted from the dataset, its manifest entry
fields store the snapshot ID in which the file was deleted and status 2
(deleted). The file may be deleted from the file system when the snapshot in
which it was deleted is garbage collected, assuming that older snapshots have
also been garbage collected [1].
-Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
-The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
+ The `sequence_number` field represents the data sequence number and must
never change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status
is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry
must include both sequence numbers explicitly.
-Notes:
+ Notes:
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
+ 1. Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected. But this is harder to
detect and requires finding the diff of multiple snapshots. It is easier to
track what files are deleted in a snapshot and delete them when that snapshot
expires. It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number`
and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3:
DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content
stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
*required* | Writer format version. V4 writers must produce `format_version` 4.
|
+ | 100 | **`location`** | `string` | *required* | *required* | Location of
the file or manifest. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* |
Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the
partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID
representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of
records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* |
*optional* | Column stats. See [Column Stats
Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* |
*optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional*
| Split offsets for the data or equality delete file. Must be sorted ascending.
|
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* |
Field ids for row equality in equality delete files. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
*optional* | Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
*optional* | Column update files associated with this entry. |
+
+ Value 1 (POSITION_DELETES) is not used in v4. Writers must not produce
`content_type` 1.
+
+ V4 leaf data manifests must only contain entries with `content_type` 0
(DATA); V4 leaf delete manifests must only contain entries with `content_type`
2 (EQUALITY DELETES). A root manifest may reference V1-V3 manifests; V1-V3 leaf
manifest references must have `format_version` set to 0.
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null
otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null
otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null
otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
Review Comment:
Probably not here with the constraints IMO. In case later on we'll have a
dedicated section for describing how we update MDVs (and maybe other stuff in
manifests) then it will give enough explanation I think.
##########
format/spec.md:
##########
@@ -691,21 +707,118 @@ The `manifest_entry` struct consists of the following
fields:
| | _optional_ | **`4 file_sequence_number`** | `long`
| File sequence number indicating
when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file`
`struct` (see below) | File path, partition tuple,
metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
+ The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
-When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
+ When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
-When a file is replaced or deleted from the dataset, its manifest entry fields
store the snapshot ID in which the file was deleted and status 2 (deleted). The
file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+ When a file is replaced or deleted from the dataset, its manifest entry
fields store the snapshot ID in which the file was deleted and status 2
(deleted). The file may be deleted from the file system when the snapshot in
which it was deleted is garbage collected, assuming that older snapshots have
also been garbage collected [1].
-Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
-The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
+ The `sequence_number` field represents the data sequence number and must
never change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status
is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry
must include both sequence numbers explicitly.
-Notes:
+ Notes:
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
+ 1. Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected. But this is harder to
detect and requires finding the diff of multiple snapshots. It is easier to
track what files are deleted in a snapshot and delete them when that snapshot
expires. It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number`
and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3:
DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content
stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
*required* | Writer format version. V4 writers must produce `format_version` 4.
|
+ | 100 | **`location`** | `string` | *required* | *required* | Location of
the file or manifest. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* |
Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the
partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID
representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of
records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* |
*optional* | Column stats. See [Column Stats
Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* |
*optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional*
| Split offsets for the data or equality delete file. Must be sorted ascending.
|
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* |
Field ids for row equality in equality delete files. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
*optional* | Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
*optional* | Column update files associated with this entry. |
+
+ Value 1 (POSITION_DELETES) is not used in v4. Writers must not produce
`content_type` 1.
+
+ V4 leaf data manifests must only contain entries with `content_type` 0
(DATA); V4 leaf delete manifests must only contain entries with `content_type`
2 (EQUALITY DELETES). A root manifest may reference V1-V3 manifests; V1-V3 leaf
manifest references must have `format_version` set to 0.
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null
otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null
otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null
otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is 3 or 4 (manifests).
+ - `split_offsets` must be null when `content_type` is 3 or 4 (manifests).
+ - `tracking.deleted_positions` and `tracking.replaced_positions` must be
null when `content_type` is not 3 or 4.
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | *required* | Used to track additions,
deletions, replacements, and modifications. When a data file's
`deletion_vector` or `column_files` change, REPLACED marks the prior version of
the entry and MODIFIED marks the new, live version. For leaf manifest entries,
MODIFIED marks a live manifest whose `dv` changed. Deletes are not used in
scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID
where the file was added or deleted. Inherited when null. Optional for leaf
manifests, required for root. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | *optional* | Snapshot
ID where the deletion vector was added. Inherited when null. Must be null when
`deletion_vector` is null. |
+ | 3 | **`sequence_number`** | `long` | *optional* | *optional* | Data
sequence number of the file. Inherited when null and status is 1 (ADDED). Must
equal `file_sequence_number` if `content_type` is 3 or 4. Optional for leaf
manifests, required for root. |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | *optional* | File
sequence number indicating when the file was added. Inherited when null and
status is ADDED. Must equal `sequence_number` if `content_type` is 3 or 4. |
+ | 142 | **`first_row_id`** | `long` | *optional* | *optional* | The
`_row_id` for the first row in the data file if `content_type` is 0. If
`content_type` is 3, this is the starting `_row_id` to assign to rows added by
ADDED data files. See [First Row ID Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were deleted in this snapshot. |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were replaced in this snapshot. |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 155 | **`location`** | `string` | *required* | *required* | Location of
the Puffin file. |
+ | 144 | **`offset`** | `long` | *required* | *required* | Offset in the
file where the content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | *required* | Length of
the referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | *required* | Cardinality
of the deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 504 | **`added_files_count`** | `long` | *required* | *required* | Count
of entries with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `long` | *required* | *required* |
Count of entries with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `long` | *required* | *required* |
Count of entries with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `long` | *required* | *required* |
Count of entries with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `long` | *required* | *required* |
Count of entries with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | *required* | Total
number of rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | *required* |
Total number of rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | *required* |
Total number of rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | *required* |
Total number of rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | *required* |
Total number of rows in MODIFIED entries. |
+ | 516 | **`min_sequence_number`** | `long` | *required* | *required* |
Minimum data sequence number of all live entries in the manifest. |
+ | 522 | **`dv`** | `binary` | *optional* | *optional* | Roaring bitmap of
entry positions in the manifest that are not live in the current snapshot. |
+ | 523 | **`dv_cardinality`** | `long` | *optional* | *optional* |
Cardinality of the manifest deletion vector. Must be set when `dv` is non-null;
must be null otherwise. |
+
+ **`column_file` struct (element 159 of `column_files`, field 158)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 161 | **`format_version`** | `int` | *required* | *required* | Format
version of this column file. |
+ | 162 | **`field_ids`** | `list<163: int>` | *required* | *required* |
Live field IDs stored in this column file. |
+ | 164 | **`location`** | `string` | *required* | *required* | Location of
the column file. |
+ | 165 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, or `parquet`. |
+ | 166 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total column file size in bytes. |
+ | 167 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 168 | **`split_offsets`** | `list<169: long>` | *optional* | *optional*
| Split offsets for the column file. Must be sorted ascending. |
+
+ When a file is added to the dataset, its content entry must set status to
ADDED (1) and store the snapshot ID in which the file was added.
+
+ When a data file's deletion vector or column files are updated, the writer
must record two content entries for the file in the snapshot: a REPLACED (3)
entry for the prior version and a MODIFIED (4) entry for the new, live version.
Both entries store the snapshot ID of the update. A MODIFIED data file entry
must always have a corresponding REPLACED entry.
Review Comment:
For data file relation between REPLACED and MODIFIED entries is symmetrical,
but this comment only expresses one direction.
##########
format/spec.md:
##########
@@ -691,21 +707,118 @@ The `manifest_entry` struct consists of the following
fields:
| | _optional_ | **`4 file_sequence_number`** | `long`
| File sequence number indicating
when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file`
`struct` (see below) | File path, partition tuple,
metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
+ The manifest entry fields are used to keep track of the snapshot in which
files were added or logically deleted. The `data_file` struct, defined below,
is nested inside the manifest entry so that it can be easily passed to job
planning without the manifest entry fields.
-When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
+ When a file is added to the dataset, its manifest entry should store the
snapshot ID in which the file was added and set status to 1 (added).
-When a file is replaced or deleted from the dataset, its manifest entry fields
store the snapshot ID in which the file was deleted and status 2 (deleted). The
file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+ When a file is replaced or deleted from the dataset, its manifest entry
fields store the snapshot ID in which the file was deleted and status 2
(deleted). The file may be deleted from the file system when the snapshot in
which it was deleted is garbage collected, assuming that older snapshots have
also been garbage collected [1].
-Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
-The `sequence_number` field represents the data sequence number and must never
change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1
(added). If the entry status is 0 (existing) or 2 (deleted), the entry must
include both sequence numbers explicitly.
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the
snapshot ID optional. Values for these fields are inherited from manifest
metadata when `null`. That is, if the field is `null` for an entry, then the
entry must inherit its value from the manifest file's metadata, stored in the
manifest list.
+ The `sequence_number` field represents the data sequence number and must
never change after a file is added to the dataset. The data sequence number
represents a relative age of the file content and should be used for planning
which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the
snapshot that added the file and must also remain unchanged upon assigning at
commit. The file sequence number can't be used for pruning delete files as the
data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status
is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry
must include both sequence numbers explicitly.
-Notes:
+ Notes:
-1. Technically, data files can be deleted when the last snapshot that contains
the file as “live” data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and
`snapshot_id` to inherit are always available.
+ 1. Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected. But this is harder to
detect and requires finding the diff of multiple snapshots. It is easier to
track what files are deleted in a snapshot and delete them when that snapshot
expires. It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number`
and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3:
DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content
stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* |
*required* | Writer format version. V4 writers must produce `format_version` 4.
|
+ | 100 | **`location`** | `string` | *required* | *required* | Location of
the file or manifest. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* |
Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the
partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID
representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of
records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* |
Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* |
*optional* | Column stats. See [Column Stats
Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* |
*optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional*
| Split offsets for the data or equality delete file. Must be sorted ascending.
|
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* |
Field ids for row equality in equality delete files. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* |
*optional* | Row-level deletion vector for a data file. |
+ | 158 | **`column_files`** | `list<159: column_file>` | *optional* |
*optional* | Column update files associated with this entry. |
+
+ Value 1 (POSITION_DELETES) is not used in v4. Writers must not produce
`content_type` 1.
+
+ V4 leaf data manifests must only contain entries with `content_type` 0
(DATA); V4 leaf delete manifests must only contain entries with `content_type`
2 (EQUALITY DELETES). A root manifest may reference V1-V3 manifests; V1-V3 leaf
manifest references must have `format_version` set to 0.
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null
otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null
otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null
otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is 3 or 4 (manifests).
+ - `split_offsets` must be null when `content_type` is 3 or 4 (manifests).
+ - `tracking.deleted_positions` and `tracking.replaced_positions` must be
null when `content_type` is not 3 or 4.
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3:
REPLACED, 4: MODIFIED) | *required* | *required* | Used to track additions,
deletions, replacements, and modifications. When a data file's
`deletion_vector` or `column_files` change, REPLACED marks the prior version of
the entry and MODIFIED marks the new, live version. For leaf manifest entries,
MODIFIED marks a live manifest whose `dv` changed. Deletes are not used in
scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID
where the file was added or deleted. Inherited when null. Optional for leaf
manifests, required for root. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | *optional* | Snapshot
ID where the deletion vector was added. Inherited when null. Must be null when
`deletion_vector` is null. |
+ | 3 | **`sequence_number`** | `long` | *optional* | *optional* | Data
sequence number of the file. Inherited when null and status is 1 (ADDED). Must
equal `file_sequence_number` if `content_type` is 3 or 4. Optional for leaf
manifests, required for root. |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | *optional* | File
sequence number indicating when the file was added. Inherited when null and
status is ADDED. Must equal `sequence_number` if `content_type` is 3 or 4. |
+ | 142 | **`first_row_id`** | `long` | *optional* | *optional* | The
`_row_id` for the first row in the data file if `content_type` is 0. If
`content_type` is 3, this is the starting `_row_id` to assign to rows added by
ADDED data files. See [First Row ID Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were deleted in this snapshot. |
+ | 7 | **`replaced_positions`** | `binary` | *optional* | *optional* |
[TODO: link to roaring bitmap spec] Roaring bitmap of positions in the
referenced leaf manifest that were replaced in this snapshot. |
+
+ **`deletion_vector` struct (field 148)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 155 | **`location`** | `string` | *required* | *required* | Location of
the Puffin file. |
+ | 144 | **`offset`** | `long` | *required* | *required* | Offset in the
file where the content starts. |
+ | 145 | **`size_in_bytes`** | `long` | *required* | *required* | Length of
the referenced content stored in the file. |
+ | 156 | **`cardinality`** | `long` | *required* | *required* | Cardinality
of the deletion vector. |
+ | 149 | **`key_metadata`** | `binary` | *optional* | *optional* |
Implementation-specific key metadata for encryption. |
+
+ **`manifest_info` struct (field 150)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 504 | **`added_files_count`** | `long` | *required* | *required* | Count
of entries with status ADDED in the manifest. |
+ | 505 | **`existing_files_count`** | `long` | *required* | *required* |
Count of entries with status EXISTING in the manifest. |
+ | 506 | **`deleted_files_count`** | `long` | *required* | *required* |
Count of entries with status DELETED in the manifest. |
+ | 520 | **`replaced_files_count`** | `long` | *required* | *required* |
Count of entries with status REPLACED in the manifest. |
+ | 524 | **`modified_files_count`** | `long` | *required* | *required* |
Count of entries with status MODIFIED in the manifest. |
+ | 512 | **`added_rows_count`** | `long` | *required* | *required* | Total
number of rows in ADDED entries. |
+ | 513 | **`existing_rows_count`** | `long` | *required* | *required* |
Total number of rows in EXISTING entries. |
+ | 514 | **`deleted_rows_count`** | `long` | *required* | *required* |
Total number of rows in DELETED entries. |
+ | 521 | **`replaced_rows_count`** | `long` | *required* | *required* |
Total number of rows in REPLACED entries. |
+ | 525 | **`modified_rows_count`** | `long` | *required* | *required* |
Total number of rows in MODIFIED entries. |
+ | 516 | **`min_sequence_number`** | `long` | *required* | *required* |
Minimum data sequence number of all live entries in the manifest. |
+ | 522 | **`dv`** | `binary` | *optional* | *optional* | Roaring bitmap of
entry positions in the manifest that are not live in the current snapshot. |
+ | 523 | **`dv_cardinality`** | `long` | *optional* | *optional* |
Cardinality of the manifest deletion vector. Must be set when `dv` is non-null;
must be null otherwise. |
+
+ **`column_file` struct (element 159 of `column_files`, field 158)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 161 | **`format_version`** | `int` | *required* | *required* | Format
version of this column file. |
+ | 162 | **`field_ids`** | `list<163: int>` | *required* | *required* |
Live field IDs stored in this column file. |
+ | 164 | **`location`** | `string` | *required* | *required* | Location of
the column file. |
+ | 165 | **`file_format`** | `string` | *required* | *required* | String
file format name: `avro`, `orc`, or `parquet`. |
Review Comment:
Do we want to include the list of supported file format as of now? Who know
if and when these change.
##########
format/spec.md:
##########
@@ -1217,7 +1315,7 @@ Statistics files metadata within `statistics` table
metadata field is a struct w
| _required_ | _required_ | **`file-size-in-bytes`** | `long`
| Size of the statistics file. |
| _required_ | _required_ | **`file-footer-size-in-bytes`** | `long`
| Total size of the statistics file's footer (not the footer payload
size). See [Puffin file format](puffin-spec.md) for footer definition. |
| _optional_ | _optional_ | **`key-metadata`** |
| Base64-encoded implementation-specific key metadata for encryption. |
- | _required_ | _required_ | **`blob-metadata`** | `list<blob
metadata>` (see below) | A list of the blob metadata for statistics contained
in the file with structure described below. |
+ | _required_ | _required_ | **`blob-metadata`** | `list<blob
metadata>` | A list of the blob metadata for statistics contained in the file
with structure described below. |
Review Comment:
Not sure if this line of change is relevant to the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
