mbutrovich commented on code in PR #17822: URL: https://github.com/apache/iceberg/pull/17822#discussion_r4125411269
########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. Review Comment: How should commit retries work with enforced constraints? If I'm reading [Commit Conflict Resolution and Retry](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1311-L1318) correctly, "append operations have no requirements and can always be applied", so two appends that each pass a `unique` check against the same parent could both commit. Should that section get rules for constraints? For example, a retry could recheck its added rows, recheck when a concurrent commit adds a constraint or sets `enforced`, and recompute `constraint-statuses` from the new parent, similar to how [`first-row-id`](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L989) is reassigned. ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). Review Comment: Is "the same restrictions" meant to cover everything identifier fields exclude? [Identifier fields](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L436) also rule out `float`, `double`, and [optional](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L237) fields, which this sentence doesn't mention. Would it be clearer to list the allowed types and nullability for each constraint type? Are `geometry` and `geography` allowed as keys? They are [primitive types](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L284-L285), so this line allows them, but as far as I can tell the spec doesn't define equality for them. The [`identity` transform](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L572) excludes them, and the same shape can have more than one WKB encoding. For `primary-key`, should the key fields also have to be `required` in the schema? That would match identifier fields, and the schema would then rule out null keys even when the constraint isn't enforced. For `unique`, a field in an optional struct reads as null when the struct is null, and the null rule already exempts those rows. Is the required-struct restriction needed there? ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. + +Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. + +Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. + +#### Check Constraint Expressions + +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. + +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. Review Comment: Does "evaluated for each row over the values of that row" rule out functions whose result depends on something other than the row, like `current_date()` or `rand()`? The list of excluded expressions covers other rows and other tables but not these. With `current_date()`, a row that passes today could fail tomorrow, so a `validated` status could become false without any write. Should a new version of a [UDF](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/udf-spec.md) that a check expression calls count as a change to the constraint? The [apply function](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/expressions-spec.md?plain=1#L85) section uses a check constraint that calls a UDF as its example, and a UDF's definition can get a [new version](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/udf-spec.md?plain=1#L142-L145) without any change to the constraint. Line 700 requires a new `constraint-id` when the expression changes, "so that statuses recorded for the old definition are not read as applying to the new one". A [function reference](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/expressions-spec.md?plain=1#L266-L269) has no way to name a UDF version today. ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. + +Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. + +Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. + +#### Check Constraint Expressions + +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. + +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. + +Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. + +To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. + +#### Constraints and Schema Evolution + +A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. + +* Renaming or reordering a referenced field is allowed; the constraint continues to apply to the same fields. +* If a dropped field is referenced only by single-column constraints, the drop is allowed and those constraints are removed automatically. If a dropped field is referenced by a multi-column constraint, the writer must reject the drop unless that constraint is removed in the same change. +* The type of a referenced field must not be changed, even for type promotions that are otherwise allowed. This restriction may be relaxed in a later version. Review Comment: Does "type" here include whether a field is required? The spec treats required and optional as a property of the [field](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L237), separate from its type, and [Appendix C](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1688) serializes `required` next to `type`. As I read it, a key field can still be made optional, and so can a struct that contains one, since the struct isn't a referenced field. Java supports both through [`UpdateSchema.makeColumnOptional`](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/api/src/main/java/org/apache/iceberg/UpdateSchema.java#L536-L542). What should happen to a `primary-key` or `unique` constraint in those cases? ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. + +Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. + +Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. + +#### Check Constraint Expressions + +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. + +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. + +Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. Review Comment: More null chat :) Is this consistent with the expressions spec? Comparisons there are [null-safe](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/expressions-spec.md?plain=1#L150-L166), so `34 != null` and `null <= null` are both true. What do you think about pointing to the [Boolean logic](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/expressions-spec.md?plain=1#L190-L192) section, which already covers the SQL `CHECK` translation? ```suggestion Iceberg predicates use two-valued logic and null-safe comparisons, as defined in the [expressions spec](expressions-spec.md#boolean-logic). This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false. ``` ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. + +Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. + +Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. + +#### Check Constraint Expressions + +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. + +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. + +Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. + +To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. + +#### Constraints and Schema Evolution + +A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. + +* Renaming or reordering a referenced field is allowed; the constraint continues to apply to the same fields. +* If a dropped field is referenced only by single-column constraints, the drop is allowed and those constraints are removed automatically. If a dropped field is referenced by a multi-column constraint, the writer must reject the drop unless that constraint is removed in the same change. +* The type of a referenced field must not be changed, even for type promotions that are otherwise allowed. This restriction may be relaxed in a later version. + +Changing a constraint's own definition is governed by the mutability rule above: only `name` and `enforced` may be changed in place; changing an `expression` or `field-ids` requires a new constraint. + +#### Constraint Validation + +Enforcement and validation are separate properties. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. + +The status of a constraint for a snapshot is one of: + +| Status | Description | +|---------------|-------------| +| `validated` | The constraint was checked and holds for all rows in the snapshot | +| `valid` | The constraint holds for all rows in the snapshot because it was enforced for the commit that produced the snapshot and the parent snapshot's status is `validated` or `valid` | +| `invalid` | The constraint was checked and at least one row in the snapshot violates it | +| `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | + +A snapshot's `constraint-statuses` records, for each status, the IDs of the constraints that have that status for the snapshot: + +| Requirement | Field name | Type | Description | +|-------------|-------------------|-------------|-------------| +| _optional_ | **`validated`** | `list<int>` | IDs of constraints that are `validated` for the snapshot | +| _optional_ | **`valid`** | `list<int>` | IDs of constraints that are `valid` for the snapshot | +| _optional_ | **`invalid`** | `list<int>` | IDs of constraints that are `invalid` for the snapshot | +| _optional_ | **`unvalidated`** | `list<int>` | IDs of constraints that are `unvalidated` for the snapshot | + +Each list contains the `constraint-id` of every constraint that has that status for the snapshot. Every constraint that exists when the snapshot is created must be listed in exactly one of the four lists, and a `constraint-id` must not appear in more than one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list did not exist when the snapshot was created, so the snapshot makes no claim about it. + +This is an explicit representation: each constraint's status is recorded independently, so the size of `constraint-statuses` grows with the number of constraints in a table. This keeps the encoding simple; more compact representations may be added in a later version if it becomes a problem. + +Readers must determine the status of a constraint for a snapshot as follows: + +1. If the snapshot has no `constraint-statuses`, the snapshot makes no claim about any constraint +2. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status +3. Otherwise, the constraint did not exist when the snapshot was created and the snapshot makes no claim about it + +Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: + +* A constraint must not be listed as `validated` unless it was checked for every row in the snapshot +* A constraint must not be listed as `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` +* A constraint must not be listed as `invalid` unless a row in the snapshot is known to violate it +* `unvalidated` is the status of a constraint that cannot be listed in any other status + +Enforcing a constraint for a commit is not sufficient to list it as `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. + +When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. + +A writer does not have to check every row in a single scan. After checking every row in an ancestor snapshot, a writer may check only the rows added between that ancestor and the current snapshot and record `validated` for the current snapshot. This allows a validation to finish on a table that is written concurrently, without blocking writes or restarting the scan. + +A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that changes only constraint statuses may reuse its parent's manifest list. Review Comment: Which [`operation`](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L962) and [`added-rows`](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L965) should a status-only snapshot use? `operation` is required, and none of the [four operations](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L968-L973) seem to fit. Reusing the parent's manifest list also seems to conflict with [Manifest Lists](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1002), which says a new manifest list is written for each commit attempt. ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. + +Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. + +Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. + +#### Check Constraint Expressions + +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. + +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. + +Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. + +To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. + +#### Constraints and Schema Evolution + +A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. + +* Renaming or reordering a referenced field is allowed; the constraint continues to apply to the same fields. +* If a dropped field is referenced only by single-column constraints, the drop is allowed and those constraints are removed automatically. If a dropped field is referenced by a multi-column constraint, the writer must reject the drop unless that constraint is removed in the same change. +* The type of a referenced field must not be changed, even for type promotions that are otherwise allowed. This restriction may be relaxed in a later version. + +Changing a constraint's own definition is governed by the mutability rule above: only `name` and `enforced` may be changed in place; changing an `expression` or `field-ids` requires a new constraint. + +#### Constraint Validation + +Enforcement and validation are separate properties. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. + +The status of a constraint for a snapshot is one of: + +| Status | Description | +|---------------|-------------| +| `validated` | The constraint was checked and holds for all rows in the snapshot | +| `valid` | The constraint holds for all rows in the snapshot because it was enforced for the commit that produced the snapshot and the parent snapshot's status is `validated` or `valid` | +| `invalid` | The constraint was checked and at least one row in the snapshot violates it | +| `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | + +A snapshot's `constraint-statuses` records, for each status, the IDs of the constraints that have that status for the snapshot: + +| Requirement | Field name | Type | Description | +|-------------|-------------------|-------------|-------------| +| _optional_ | **`validated`** | `list<int>` | IDs of constraints that are `validated` for the snapshot | +| _optional_ | **`valid`** | `list<int>` | IDs of constraints that are `valid` for the snapshot | +| _optional_ | **`invalid`** | `list<int>` | IDs of constraints that are `invalid` for the snapshot | +| _optional_ | **`unvalidated`** | `list<int>` | IDs of constraints that are `unvalidated` for the snapshot | + +Each list contains the `constraint-id` of every constraint that has that status for the snapshot. Every constraint that exists when the snapshot is created must be listed in exactly one of the four lists, and a `constraint-id` must not appear in more than one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list did not exist when the snapshot was created, so the snapshot makes no claim about it. + +This is an explicit representation: each constraint's status is recorded independently, so the size of `constraint-statuses` grows with the number of constraints in a table. This keeps the encoding simple; more compact representations may be added in a later version if it becomes a problem. + +Readers must determine the status of a constraint for a snapshot as follows: + +1. If the snapshot has no `constraint-statuses`, the snapshot makes no claim about any constraint +2. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status +3. Otherwise, the constraint did not exist when the snapshot was created and the snapshot makes no claim about it + +Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: + +* A constraint must not be listed as `validated` unless it was checked for every row in the snapshot +* A constraint must not be listed as `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` Review Comment: Should [compaction](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L971) and deletes keep their parent's status? Neither can introduce a violation, but as I read these rules, they downgrade the status unless the writer rechecks every row. A `validated` status would become `valid` for an enforced constraint and `unvalidated` otherwise. Could a commit that adds no new rows carry its parent's status forward? ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. + +Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. + +Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. + +#### Check Constraint Expressions + +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. + +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. + +Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. + +To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. + +#### Constraints and Schema Evolution + +A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. + +* Renaming or reordering a referenced field is allowed; the constraint continues to apply to the same fields. +* If a dropped field is referenced only by single-column constraints, the drop is allowed and those constraints are removed automatically. If a dropped field is referenced by a multi-column constraint, the writer must reject the drop unless that constraint is removed in the same change. +* The type of a referenced field must not be changed, even for type promotions that are otherwise allowed. This restriction may be relaxed in a later version. + +Changing a constraint's own definition is governed by the mutability rule above: only `name` and `enforced` may be changed in place; changing an `expression` or `field-ids` requires a new constraint. + +#### Constraint Validation + +Enforcement and validation are separate properties. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. + +The status of a constraint for a snapshot is one of: + +| Status | Description | +|---------------|-------------| +| `validated` | The constraint was checked and holds for all rows in the snapshot | +| `valid` | The constraint holds for all rows in the snapshot because it was enforced for the commit that produced the snapshot and the parent snapshot's status is `validated` or `valid` | +| `invalid` | The constraint was checked and at least one row in the snapshot violates it | +| `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | + +A snapshot's `constraint-statuses` records, for each status, the IDs of the constraints that have that status for the snapshot: + +| Requirement | Field name | Type | Description | +|-------------|-------------------|-------------|-------------| +| _optional_ | **`validated`** | `list<int>` | IDs of constraints that are `validated` for the snapshot | +| _optional_ | **`valid`** | `list<int>` | IDs of constraints that are `valid` for the snapshot | +| _optional_ | **`invalid`** | `list<int>` | IDs of constraints that are `invalid` for the snapshot | +| _optional_ | **`unvalidated`** | `list<int>` | IDs of constraints that are `unvalidated` for the snapshot | + +Each list contains the `constraint-id` of every constraint that has that status for the snapshot. Every constraint that exists when the snapshot is created must be listed in exactly one of the four lists, and a `constraint-id` must not appear in more than one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list did not exist when the snapshot was created, so the snapshot makes no claim about it. + +This is an explicit representation: each constraint's status is recorded independently, so the size of `constraint-statuses` grows with the number of constraints in a table. This keeps the encoding simple; more compact representations may be added in a later version if it becomes a problem. + +Readers must determine the status of a constraint for a snapshot as follows: + +1. If the snapshot has no `constraint-statuses`, the snapshot makes no claim about any constraint +2. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status +3. Otherwise, the constraint did not exist when the snapshot was created and the snapshot makes no claim about it + +Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: + +* A constraint must not be listed as `validated` unless it was checked for every row in the snapshot +* A constraint must not be listed as `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` +* A constraint must not be listed as `invalid` unless a row in the snapshot is known to violate it +* `unvalidated` is the status of a constraint that cannot be listed in any other status + +Enforcing a constraint for a commit is not sufficient to list it as `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. + +When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. + +A writer does not have to check every row in a single scan. After checking every row in an ancestor snapshot, a writer may check only the rows added between that ancestor and the current snapshot and record `validated` for the current snapshot. This allows a validation to finish on a table that is written concurrently, without blocking writes or restarting the scan. + +A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that changes only constraint statuses may reuse its parent's manifest list. + +A constraint that a snapshot reports as `valid` may later be found not to hold for that snapshot. A `valid` status depends on every commit in the snapshot's history having correctly enforced the constraint, and Iceberg records the status a writer reports without re-checking the data. So if any of those writers was buggy or non-compliant, the `valid` status can be wrong even though nothing detected it at commit time. A `validated` status, which reflects an actual scan of all rows, does not depend on that chain. Writers should then commit a snapshot that records `invalid` and should expire the snapshots that state that the constraint holds, because queries against those snapshots would otherwise continue to rely on a constraint that does not hold. Review Comment: Does this conflict with line 775? Expiring the snapshots that "state that the constraint holds" would include the `validated` ones, but line 775 keeps `validated` separate so the latest one can be found. Expiring also removes time travel, and the [retention rules](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1123-L1131) keep [tagged](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1099-L1103) snapshots anyway. Should this stop at committing a snapshot that records `invalid`? ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list<int>` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | + +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. Review Comment: The [Flink sink](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/SinkUtil.java#L60) uses [identifier fields](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L430-L436) as its default equality fields (the columns it matches on when it writes [equality deletes](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1417-L1425) for upserts), so I think this changes behavior for upgraded tables. Should the upgrade spell out a few details so implementations agree? - Which schema's `identifier-field-ids` are used, since they can differ across schemas? - What `constraint-id`, `name`, `timestamp-ms`, and `last-constraint-id` does the upgrade write? - Should v4 writers remove `identifier-field-ids`, and what should v4 readers do if they find them? - Line 720 would block `int` to `long` [type promotion](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L350) on former identifier fields. Is that intended? Should the [Identifier Field IDs](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L430-L436) section and [Appendix C](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1656) also mention that v4 doesn't use them? And would this part be easier to review as its own PR? ########## format/spec.md: ########## @@ -654,6 +655,127 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +Constraints are added in v4 and are not supported in v3 or earlier. + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value Review Comment: I know we discussed nulls a bit in the calls about this, but going to revisit the topic :) For a multi-column key, is a row exempt when any key field is null, or only when all of them are? For example, with a key on `(a, b)`, can two rows both have `a = 1` and `b = null`? SQL `UNIQUE` allows it, but I could also read "the non-null values of a set of fields" as comparing `a` alone, which would make those rows duplicates. Which equality should keys use for floats? As far as I can tell, the spec handles signed zero differently in different places. [Partition equality](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1096) keeps `-0.0` and `0.0` distinct, while the [hash definition](https://github.com/apache/iceberg/blob/e689699fa6392a2da9aa10870f0d4de575ae23c1/format/spec.md?plain=1#L1654) maps `-0.0` to `0.0`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
