stevenzwu commented on code in PR #18526:
URL: https://github.com/apache/iceberg/pull/18526#discussion_r4463420417
##########
format/spec.md:
##########
@@ -522,15 +522,17 @@ The set of metadata columns is:
| **`2147483541 _commit_snapshot_id`** | `long` | The
snapshot ID in which the change occurred
|
| **`2147483540 _row_id`** | `long` | A unique
long assigned for row lineage, see [Row Lineage](#row-lineage)
|
| **`2147483539 _last_updated_sequence_number`** | `long` | The
sequence number which last updated this row, see [Row Lineage](#row-lineage)
|
+| **`2147483538 _last_updated_time`** | `timestamptz` | The
timestamp of the snapshot which last updated this row (v4 and later), see [Row
Lineage](#row-lineage) |
#### Row Lineage
In v3 and later, an Iceberg table must track row lineage fields for all newly
created rows. Engines must maintain the `next-row-id` table field and the
following row-level fields when writing data files:
* `_row_id` a unique long identifier for every row within the table. The value
is assigned via inheritance when a row is first added to the table.
* `_last_updated_sequence_number` the sequence number of the commit that last
updated a row. The value is inherited when a row is first added or modified.
+* `_last_updated_time` (v4 and later) the timestamp of the snapshot that last
updated a row. The value is inherited when a row is first added or modified.
Review Comment:
Removed "v4 and later" from the other places, including the reserved field
ID table, and kept it only here.
##########
format/spec.md:
##########
@@ -522,15 +522,17 @@ The set of metadata columns is:
| **`2147483541 _commit_snapshot_id`** | `long` | The
snapshot ID in which the change occurred
|
| **`2147483540 _row_id`** | `long` | A unique
long assigned for row lineage, see [Row Lineage](#row-lineage)
|
| **`2147483539 _last_updated_sequence_number`** | `long` | The
sequence number which last updated this row, see [Row Lineage](#row-lineage)
|
+| **`2147483538 _last_updated_time`** | `timestamptz` | The
timestamp of the snapshot which last updated this row (v4 and later), see [Row
Lineage](#row-lineage) |
Review Comment:
This also matches the `snapshots.committed_at` and `history.made_current_at`
metadata table columns, which are `timestamptz`.
##########
format/spec.md:
##########
@@ -2300,6 +2320,15 @@ Constraints are added in v4:
* Snapshots may contain `constraint-statuses`
* Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids`
as a `primary-key` constraint that is not enforced, unless a field cannot be
used in a `primary-key` constraint
+Row lineage changes:
+
+* When writing a manifest, ADDED entries and entries MODIFIED by column files
must be written with a null `snapshot_timestamp`
+* When an entry has a null `snapshot_timestamp`, readers must assign it the
snapshot's `timestamp-ms` converted to microseconds in the root manifest, or
the leaf manifest's `snapshot_timestamp` in a leaf manifest
Review Comment:
Reduced this to two bullets noting that `snapshot_timestamp` and
`_last_updated_timestamp` were added.
##########
format/spec.md:
##########
@@ -1237,6 +1243,20 @@ Inheriting sequence numbers through the metadata tree
allows writing a new manif
Inheritance does not apply to v1, which does not have inherited fields like
sequence number. When reading v1 manifests with no sequence number column,
sequence numbers for all files must default to 0.
+##### Snapshot Timestamp Inheritance
+
+In v4 and later, the `snapshot_timestamp` tracking field records the
`timestamp-ms` of the snapshot that added or last modified a file's data,
converted to microseconds. It is used to assign `_last_updated_time` for [row
lineage](#row-lineage).
+
+Writers must set `tracking.snapshot_timestamp` for data file and leaf manifest
entries as follows:
Review Comment:
Changed to "When writing a manifest, writers must set `snapshot_timestamp`
as follows:"
##########
format/spec.md:
##########
@@ -522,15 +522,17 @@ The set of metadata columns is:
| **`2147483541 _commit_snapshot_id`** | `long` | The
snapshot ID in which the change occurred
|
| **`2147483540 _row_id`** | `long` | A unique
long assigned for row lineage, see [Row Lineage](#row-lineage)
|
| **`2147483539 _last_updated_sequence_number`** | `long` | The
sequence number which last updated this row, see [Row Lineage](#row-lineage)
|
+| **`2147483538 _last_updated_time`** | `timestamptz` | The
timestamp of the snapshot which last updated this row (v4 and later), see [Row
Lineage](#row-lineage) |
Review Comment:
Renamed to `_last_updated_timestamp`.
##########
format/spec.md:
##########
@@ -1237,6 +1243,20 @@ Inheriting sequence numbers through the metadata tree
allows writing a new manif
Inheritance does not apply to v1, which does not have inherited fields like
sequence number. When reading v1 manifests with no sequence number column,
sequence numbers for all files must default to 0.
+##### Snapshot Timestamp Inheritance
+
+In v4 and later, the `snapshot_timestamp` tracking field records the
`timestamp-ms` of the snapshot that added or last modified a file's data,
converted to microseconds. It is used to assign `_last_updated_time` for [row
lineage](#row-lineage).
+
+Writers must set `tracking.snapshot_timestamp` for data file and leaf manifest
entries as follows:
+
+* An ADDED entry must have a `null` `snapshot_timestamp`.
+* An entry MODIFIED by column files must have a `null` `snapshot_timestamp`.
+* Any other entry written to a new manifest must have a non-null
`snapshot_timestamp` (inherited or previously persisted).
+
+When reading, `snapshot_timestamp` is inherited by replacing `null` with the
snapshot's `timestamp-ms` converted to microseconds for entries in the root
manifest, or with the `snapshot_timestamp` of the root manifest entry that
tracks the leaf manifest for entries in a leaf manifest.
+
+Manifest lists and manifests written before a table was upgraded to v4 do not
have a `snapshot_timestamp` field and the field should be read as if it is
`null`. When reading a manifest list written before v4, each manifest inherits
the snapshot's `timestamp-ms` converted to microseconds, and entries in a
manifest written before v4 inherit that manifest's `snapshot_timestamp`. As a
result, rows in the table at the time of the upgrade have a
`_last_updated_time` equal to the `timestamp-ms` of the last snapshot before
the upgrade, converted to microseconds.
Review Comment:
Moved to "Row Lineage for Upgraded Tables".
##########
format/spec.md:
##########
@@ -1093,6 +1098,7 @@ IDs in the range `10_000` (inclusive) to `200_000_000`
(exclusive) are reserved
| Reserved field | ID | `base-id` | Range end |
|---------------------------------|------------|-----------|-----------|
+| `_last_updated_time` | 2147483538 | 8800 | 8999 |
Review Comment:
I wasn't sure if this should grow downwards or backwards. agree with you
that we should fill out the 9000 space first.
##########
format/spec.md:
##########
@@ -1237,6 +1243,20 @@ Inheriting sequence numbers through the metadata tree
allows writing a new manif
Inheritance does not apply to v1, which does not have inherited fields like
sequence number. When reading v1 manifests with no sequence number column,
sequence numbers for all files must default to 0.
+##### Snapshot Timestamp Inheritance
+
+In v4 and later, the `snapshot_timestamp` tracking field records the
`timestamp-ms` of the snapshot that added or last modified a file's data,
converted to microseconds. It is used to assign `_last_updated_time` for [row
lineage](#row-lineage).
+
+Writers must set `tracking.snapshot_timestamp` for data file and leaf manifest
entries as follows:
+
+* An ADDED entry must have a `null` `snapshot_timestamp`.
+* An entry MODIFIED by column files must have a `null` `snapshot_timestamp`.
Review Comment:
Combined into a single bullet.
##########
format/spec.md:
##########
@@ -1237,6 +1243,20 @@ Inheriting sequence numbers through the metadata tree
allows writing a new manif
Inheritance does not apply to v1, which does not have inherited fields like
sequence number. When reading v1 manifests with no sequence number column,
sequence numbers for all files must default to 0.
+##### Snapshot Timestamp Inheritance
+
+In v4 and later, the `snapshot_timestamp` tracking field records the
`timestamp-ms` of the snapshot that added or last modified a file's data,
converted to microseconds. It is used to assign `_last_updated_time` for [row
lineage](#row-lineage).
+
+Writers must set `tracking.snapshot_timestamp` for data file and leaf manifest
entries as follows:
+
+* An ADDED entry must have a `null` `snapshot_timestamp`.
+* An entry MODIFIED by column files must have a `null` `snapshot_timestamp`.
+* Any other entry written to a new manifest must have a non-null
`snapshot_timestamp` (inherited or previously persisted).
+
+When reading, `snapshot_timestamp` is inherited by replacing `null` with the
snapshot's `timestamp-ms` converted to microseconds for entries in the root
manifest, or with the `snapshot_timestamp` of the root manifest entry that
tracks the leaf manifest for entries in a leaf manifest.
Review Comment:
Updated to say a `null` value is inherited from its parent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
