stevenzwu commented on code in PR #13879:
URL: https://github.com/apache/iceberg/pull/13879#discussion_r3502072570
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -1051,6 +1051,15 @@ paths:
table. The configuration key "token" is used to pass an access token
to be used as a bearer token
for table requests. Otherwise, a token may be passed using a RFC 8693
token type as a configuration
key. For example, "urn:ietf:params:oauth:token-type:jwt=<JWT-token>".
+
+
+ The response may include a read-restrictions field. A client should
support this field;
+ if it does, it must fail any read against the loaded table when it
cannot apply a returned
+ restriction. This includes unrecognized action or expression types,
and actions whose
+ definition cannot be loaded, parsed, or evaluated. These restrictions
apply to every read
+ performed using this response (including subsequent planTableScan and
fetchScanTasks
+ calls), only to the authenticated principal associated with the
request, and must not be
+ interpreted as global policy.
Review Comment:
Two nits on this paragraph: the wording is a bit awkward, and it uses
"client" while the `ReadRestrictions` schema body below (line 3680+) uses
"reader" for the same actor. Suggested rewrite tightens both:
```suggestion
The response may include a `read-restrictions` field. A reader that
supports read
restrictions must fail any read against the loaded table that cannot
apply a returned
restriction in full. This includes unrecognized action or expression
types, and actions
whose definition cannot be loaded, parsed, or evaluated. These
restrictions apply to every
read performed using this response (including subsequent
planTableScan and fetchScanTasks
calls), only to the authenticated principal associated with the
request, and must not be
interpreted as global policy.
```
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3663,6 +3672,305 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader cannot apply any returned restriction (a filter
expression
+ or an action), it must fail the query and must not silently return
raw,
+ partial, or empty results.
+
+ If a projection targets a nested-typed field (struct, list, or map),
+ other projections in the same ReadRestrictions must not target any
+ descendant field-id (struct subfields, list elements, or map
keys/values)
+ at any depth. This avoids ambiguity about which action governs a
given
+ leaf value.
+
+ An empty ReadRestrictions object (no required-column-projections and
no
+ required-row-filter) imposes no restrictions and is equivalent to the
+ field being absent from the response.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ term: region
+ value: US
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original
field-id. For
+ example, if the action for field-id `9` is mask-alphanum, the
reader must
+ return the masked value as field-id `9` in the query output.
+
+ 3. A column must appear at most once in
required-column-projections.
+
+ 4. A projection must not target a map's key field-id. Applying an
action
+ to keys can produce duplicate or null keys, which readers
silently
+ coalesce or reject, causing data loss.
+ type: array
+ items:
+ $ref: '#/components/schemas/Action'
+ required-row-filter:
+ description: >
+ An expression that filters rows in the table that the
authenticated principal does not have access to.
+
+ 1. The expression must evaluate to a boolean (TRUE or FALSE;
Iceberg predicates
+ never produce NULL). A reader must discard any row for which the
filter
+ evaluates to FALSE, and no information derived from discarded
rows may be
+ included in the query result.
+
+ 2. If this property is absent, null, or always true then no
mandatory filtering is required.
+ allOf:
+ - $ref: '#/components/schemas/Expression'
+
+ Action:
+ discriminator:
+ propertyName: action
+ mapping:
+ mask-alphanum: '#/components/schemas/MaskAlphanum'
+ mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+ replace-with-null: '#/components/schemas/ReplaceWithNull'
+ show-first-4: '#/components/schemas/ShowFirst4'
+ show-last-4: '#/components/schemas/ShowLast4'
+ truncate-to-year: '#/components/schemas/TruncateToYear'
+ truncate-to-month: '#/components/schemas/TruncateToMonth'
+ sha-256-global: '#/components/schemas/Sha256Global'
+ sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+ type: object
+ required:
+ - action
+ - field-id
+ properties:
+ action:
+ type: string
+ field-id:
+ type: integer
+ description: Field ID of the column being projected.
Review Comment:
A `loadTable` response carries the full table metadata with multiple
historical schemas. Should we clarify which schema was referenced?
> Field ID in the schema identified by `current-schema-id` of the table
metadata in this response. Catalogs must not return projections referencing
field IDs that are not present in that schema.
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3663,6 +3672,305 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader cannot apply any returned restriction (a filter
expression
+ or an action), it must fail the query and must not silently return
raw,
+ partial, or empty results.
+
+ If a projection targets a nested-typed field (struct, list, or map),
+ other projections in the same ReadRestrictions must not target any
+ descendant field-id (struct subfields, list elements, or map
keys/values)
+ at any depth. This avoids ambiguity about which action governs a
given
+ leaf value.
+
+ An empty ReadRestrictions object (no required-column-projections and
no
+ required-row-filter) imposes no restrictions and is equivalent to the
+ field being absent from the response.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ term: region
+ value: US
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original
field-id. For
+ example, if the action for field-id `9` is mask-alphanum, the
reader must
+ return the masked value as field-id `9` in the query output.
+
+ 3. A column must appear at most once in
required-column-projections.
+
+ 4. A projection must not target a map's key field-id. Applying an
action
+ to keys can produce duplicate or null keys, which readers
silently
+ coalesce or reject, causing data loss.
+ type: array
+ items:
+ $ref: '#/components/schemas/Action'
+ required-row-filter:
+ description: >
+ An expression that filters rows in the table that the
authenticated principal does not have access to.
+
+ 1. The expression must evaluate to a boolean (TRUE or FALSE;
Iceberg predicates
+ never produce NULL). A reader must discard any row for which the
filter
+ evaluates to FALSE, and no information derived from discarded
rows may be
+ included in the query result.
+
+ 2. If this property is absent, null, or always true then no
mandatory filtering is required.
+ allOf:
+ - $ref: '#/components/schemas/Expression'
+
+ Action:
+ discriminator:
+ propertyName: action
+ mapping:
+ mask-alphanum: '#/components/schemas/MaskAlphanum'
+ mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+ replace-with-null: '#/components/schemas/ReplaceWithNull'
+ show-first-4: '#/components/schemas/ShowFirst4'
+ show-last-4: '#/components/schemas/ShowLast4'
+ truncate-to-year: '#/components/schemas/TruncateToYear'
+ truncate-to-month: '#/components/schemas/TruncateToMonth'
+ sha-256-global: '#/components/schemas/Sha256Global'
+ sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+ type: object
+ required:
+ - action
+ - field-id
+ properties:
+ action:
+ type: string
+ field-id:
+ type: integer
+ description: Field ID of the column being projected.
+
+ MaskAlphanum:
+ description: >
+ Redacts the column value using the following rules to transform
Unicode code points:
+
+ - Digits (U+0030–U+0039, 0-9) are replaced with 'n'
+ - The following punctuation characters are kept as-is:
+ U+0028 '(' LEFT PARENTHESIS
+ U+0029 ')' RIGHT PARENTHESIS
+ U+002C ',' COMMA
+ U+002E '.' FULL STOP
+ U+002D '-' HYPHEN-MINUS
+ U+0040 '@' COMMERCIAL AT
+ - All other Unicode characters (including letters, whitespace, and any
punctuation
+ not listed above) are replaced with 'x'
+
+ For example: "[email protected]" -> "[email protected]"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-alphanum"
+
+ MaskToFixedValue:
+ description: >
+ Replaces the column value with a type-specific fixed value.
+ Readers must use exactly the values listed below to ensure consistency
+ across implementations.
+
+ Fixed values by type:
+ - boolean: false
+ - int: 0
+ - long: 0
+ - float: 0.0
+ - double: 0.0
+ - decimal(p, s): 0 (the unscaled value is 0)
+ - string: "XXXXXXXX"
+ - date: 1970-01-01
+ - time: 00:00:00
+ - timestamp: 1970-01-01T00:00:00
+ - timestamptz: 1970-01-01T00:00:00+00:00
+ - timestamp_ns: 1970-01-01T00:00:00.000000000
+ - timestamptz_ns: 1970-01-01T00:00:00.000000000+00:00
+ - uuid: 00000000-0000-0000-0000-000000000000
+ - fixed(n): n zero bytes
+ - binary: empty byte sequence
+ - variant: {}
+ - list: empty list []
+ - map: empty map {}
+ - struct: struct with each field set to its type-specific default
(applied recursively)
Review Comment:
What happens when a struct contains a `geometry`, `geography`, or `unknown`
field? Line 3830 excludes those three types — and no fixed value is defined for
them, so the recursive rule has nowhere to land. Maybe state the following in
line 3830?
```
A catalog server must not return `mask-to-fixed-value` on a container that
recursively contains any excluded type.
```
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3480,6 +3480,309 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table, including column projections and row
filter expressions.
+
+ A client MUST enforce the restrictions defined in this object when
reading data
+ from the table.
+
+ These restrictions apply only to the authenticated principal, user,
or account
+ associated with the request. They MUST NOT be interpreted as global
policy and
+ MUST NOT be applied beyond the entity identified by the
Authentication header
+ (or other applicable authentication mechanism).
+
+ If both properties are absent or empty, the ReadRestrictions object
imposes no
+ restrictions and is equivalent to the field being absent from the
response.
+ A server MUST NOT return an action for a column whose type is not
listed in
+ that action's "Applicable to" set.
+ For all actions, if the input column value is NULL, the output MUST
be NULL.
+
+ If a column projection targets a struct-typed field, other column
projections
+ in the same ReadRestrictions MUST NOT target any of that struct's
subfields
+ (at any depth). This avoids ambiguity about which action governs a
given
+ leaf value.
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+
+ If this property is absent, a reader MAY access all columns of the
table as-is
+ without any mandatory transformations.
+
+ If this property is present, each listed column MUST have its
specified
+ action applied. Columns not listed in required-column-projections
+ are not subject to any read restrictions.
+
+ When this list is present:
+
+ 1. For each column listed in required-column-projections, the
reader MUST apply
+ the specified action before returning values for that column.
+
+ 2. The reader MUST replace all output references to the column
with the result
+ of the action, presenting the result under the original column
name. For
+ example, if the action for column cc is mask-alphanum, the
reader MUST
+ return the masked value as cc in the query output.
+
+ 3. Columns not listed in required-column-projections MAY be
projected normally
+ by the reader without any mandatory transformations.
+
+ 4. A column MUST appear at most once in
required-column-projections.
+
+ 5. If a projected column's action cannot be evaluated by the reader
+ (including unrecognized action types), the reader MUST fail
rather than
+ ignore or skip the action.
+
+ 6. Each action defines the output type for its column. For all
predefined
+ actions except apply-expression, the output type matches the
input column
+ type. For apply-expression, the output type is determined by the
expression.
+
+ type: array
+ items:
+ $ref: '#/components/schemas/Action'
+ required-row-filter:
+ description: >
+ An expression that filters rows in the table that the
authenticated principal does not have access to.
+
+ 1. The expression MUST evaluate to a boolean. A reader MUST
discard any row for which
+ the filter evaluates to FALSE, and no information derived from
discarded rows
+ MAY be included in the query result.
+
+ 2. Row filters MUST be evaluated against the original,
untransformed column values.
+ Required projections MUST be applied only after row filters are
applied.
+
+ 3. If a client cannot interpret or evaluate a provided filter
expression, it MUST fail.
+
+ 4. If this property is absent, null, or always true then no
mandatory filtering is required.
+ $ref: '#/components/schemas/Expression'
+
+ Action:
+ discriminator:
+ propertyName: action
+ mapping:
+ mask-alphanum: '#/components/schemas/MaskAlphanum'
+ mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+ replace-with-null: '#/components/schemas/ReplaceWithNull'
+ show-first-4: '#/components/schemas/ShowFirst4'
+ show-last-4: '#/components/schemas/ShowLast4'
+ truncate-to-year: '#/components/schemas/TruncateToYear'
+ truncate-to-month: '#/components/schemas/TruncateToMonth'
+ sha-256-global: '#/components/schemas/Sha256Global'
+ sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+ apply-expression: '#/components/schemas/ApplyExpression'
+ type: object
+ required:
+ - action
+ - field-id
+ properties:
+ action:
+ type: string
+ field-id:
+ type: integer
+ description: field id of the column being projected.
+
+ MaskAlphanum:
+ description: >
+ Redacts the column value Unicode code point by code point using the
following rules:
+
+ - Digits (U+0030–U+0039, 0-9) are replaced with 'n'
+ - The following punctuation characters are kept as-is:
+ U+0028 '(' LEFT PARENTHESIS
+ U+0029 ')' RIGHT PARENTHESIS
+ U+002C ',' COMMA
+ U+002E '.' FULL STOP
+ U+002D '-' HYPHEN-MINUS
+ U+0040 '@' COMMERCIAL AT
+ - All other Unicode characters (including letters, whitespace, and any
punctuation
+ not listed above) are replaced with 'x'
+
+ For example: "[email protected]" → "[email protected]"
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-alphanum"
+
+ MaskToFixedValue:
+ description: >
+ Replaces the column value with a predefined type-specific fixed value.
+ Engines MUST use exactly the values listed below to ensure consistency
+ across implementations.
+
+ Fixed values by type:
+ - boolean: false
+ - int: 0
+ - long: 0
+ - float: 0.0
+ - double: 0.0
+ - decimal(p, s): 0 (zero with s digits after the decimal point, e.g.
0.00 for decimal(p,2))
+ - string: "XXXXXXXX"
Review Comment:
Reopening to flag a few concerns the thread didn't engage. What other
systems use as the default string constant for the equivalent "replace with
fixed value" masking:
| System | Default string constant | Source |
| --- | --- | --- |
| SQL Server Dynamic Data Masking `default()` | `"XXXX"` (4 X's, or fewer if
column narrower) |
[define-a-dynamic-data-mask](https://learn.microsoft.com/en-us/sql/relational-databases/security/dynamic-data-masking#define-a-dynamic-data-mask)
|
| BigQuery Default masking value | `""` (empty string) |
[column-data-masking-intro#masking_options](https://cloud.google.com/bigquery/docs/column-data-masking-intro#masking_options)
|
| Oracle `DBMS_REDACT` FULL redaction | `" "` (single space) | [Oracle Data
Redaction
3.1](https://docs.oracle.com/cd/E11882_01/network.112/e40393/redaction.htm) |
| Snowflake masking policy | user-defined; docs example uses `'********'` |
[CREATE MASKING POLICY
example](https://docs.snowflake.com/en/sql-reference/sql/create-masking-policy#example-normal-masking-policy)
|
| PostgreSQL Anonymizer `MASKED WITH VALUE` | user-defined; docs example
uses `'CONFIDENTIAL'` |
[Destruction](https://postgresql-anonymizer.readthedocs.io/en/latest/masking_functions/#destruction)
|
Of the three systems that ship a mandated default, none use 8 X's: SQL
Server picks 4 X's, BigQuery picks empty, Oracle picks a single space. The
specific choice of `"XXXXXXXX"` has no precedent.
Other concerns:
1. **Breaks the spec's own internal pattern.** Every other fixed value in
this action is the type's empty/minimum form: `int: 0`, `binary: empty byte
sequence`, `list: empty list`, `map: empty map`, `variant: {}`. The matching
minimum for `string` is `""`. `"XXXXXXXX"` is the only entry that violates the
convention.
2. **The mask-alphanum analogy isn't quite applicable here.**
`mask-alphanum` preserves length and replaces characters in place — that's
where the familiar X-pattern comes from. `mask-to-fixed-value` is the "replace
entirely with a constant" mode, a different design. Users who want X-pattern
masking already have `mask-alphanum`.
Suggest `string: ""` — matches BigQuery, matches this action's per-type
empty/collection pattern.
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3663,6 +3672,305 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader cannot apply any returned restriction (a filter
expression
+ or an action), it must fail the query and must not silently return
raw,
+ partial, or empty results.
+
+ If a projection targets a nested-typed field (struct, list, or map),
+ other projections in the same ReadRestrictions must not target any
+ descendant field-id (struct subfields, list elements, or map
keys/values)
+ at any depth. This avoids ambiguity about which action governs a
given
+ leaf value.
+
+ An empty ReadRestrictions object (no required-column-projections and
no
+ required-row-filter) imposes no restrictions and is equivalent to the
+ field being absent from the response.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ term: region
+ value: US
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original
field-id. For
+ example, if the action for field-id `9` is mask-alphanum, the
reader must
+ return the masked value as field-id `9` in the query output.
+
+ 3. A column must appear at most once in
required-column-projections.
Review Comment:
Rule 3 doesn't pin the actor. Worth stating both sides explicitly:
```suggestion
3. A server must not return more than one projection for the
same field-id
in required-column-projections. If a duplicate field-id
appears, the reader
must fail the query.
```
(The schema can't enforce this on its own — `uniqueItems: true` tests
whole-object equality, so two entries with the same `field-id` and different
`action`s would still pass.)
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3663,6 +3672,305 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader cannot apply any returned restriction (a filter
expression
+ or an action), it must fail the query and must not silently return
raw,
+ partial, or empty results.
+
+ If a projection targets a nested-typed field (struct, list, or map),
+ other projections in the same ReadRestrictions must not target any
+ descendant field-id (struct subfields, list elements, or map
keys/values)
Review Comment:
Wording: "descendant" isn't used elsewhere in the Iceberg spec for this
relationship. `format/spec.md` consistently says "nested under", "nested in",
"child field", or "contained in" (lines 382, 413, 417, 434, 556). "Descendant"
reads correctly for tree-terminology folks but feels slightly off against the
rest of the spec. Consider:
```suggestion
If a projection targets a nested-typed field (struct, list, or
map),
other projections in the same ReadRestrictions must not target any
nested field-id (struct subfields, list elements, or map
keys/values)
at any depth. This avoids ambiguity about which action governs a
given
leaf value.
```
The parenthetical already explains the relationship, so the noun just needs
to match the rest of the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
