sungwy commented on code in PR #13879:
URL: https://github.com/apache/iceberg/pull/13879#discussion_r3462026174
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3632,6 +3641,301 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader cannot apply any returned restriction (a filter
expression
+ or an action), it must fail the query and must not silently return
raw,
+ partial, or empty results.
+
+ If a projection targets a field, that action governs every value
reachable
+ through it; other projections in the same object that target a
descendant
+ of that field have no effect.
+
+ An empty ReadRestrictions object (no required-column-projections and
no
+ required-row-filter) imposes no restrictions and is equivalent to the
+ field being absent from the response.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ term: region
+ value: US
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original column
name. For
+ example, if the action for column 'cc' is mask-alphanum, the
reader must
+ return the masked value as 'cc' in the query output.
+
+ 3. A column must appear at most once in
required-column-projections.
+ type: array
+ items:
+ $ref: '#/components/schemas/Action'
+ required-row-filter:
+ description: >
+ An expression that filters rows in the table that the
authenticated principal does not have access to.
+
+ 1. The expression must evaluate to a boolean (TRUE or FALSE;
Iceberg expressions
+ never produce NULL). A reader must discard any row for which the
filter
+ evaluates to FALSE, and no information derived from discarded
rows may be
+ included in the query result.
+
+ 2. If this property is absent, null, or always true then no
mandatory filtering is required.
+ allOf:
+ - $ref: '#/components/schemas/Expression'
+
+ Action:
+ discriminator:
+ propertyName: action
+ mapping:
+ mask-alphanum: '#/components/schemas/MaskAlphanum'
+ mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+ replace-with-null: '#/components/schemas/ReplaceWithNull'
+ show-first-4: '#/components/schemas/ShowFirst4'
+ show-last-4: '#/components/schemas/ShowLast4'
+ truncate-to-year: '#/components/schemas/TruncateToYear'
+ truncate-to-month: '#/components/schemas/TruncateToMonth'
+ sha-256-global: '#/components/schemas/Sha256Global'
+ sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+ type: object
+ required:
+ - action
+ - field-id
+ properties:
+ action:
+ type: string
+ field-id:
+ type: integer
+ description: Field ID of the column being projected.
+
+ MaskAlphanum:
+ description: >
+ Redacts the column value using the following rules to transform
Unicode code points:
+
+ - Digits (U+0030–U+0039, 0-9) are replaced with 'n'
+ - The following punctuation characters are kept as-is:
+ U+0028 '(' LEFT PARENTHESIS
+ U+0029 ')' RIGHT PARENTHESIS
+ U+002C ',' COMMA
+ U+002E '.' FULL STOP
+ U+002D '-' HYPHEN-MINUS
+ U+0040 '@' COMMERCIAL AT
+ - All other Unicode characters (including letters, whitespace, and any
punctuation
+ not listed above) are replaced with 'x'
+
+ For example: "[email protected]" -> "[email protected]"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-alphanum"
+
+ MaskToFixedValue:
+ description: >
+ Replaces the column value with a type-specific fixed value.
+ Readers must use exactly the values listed below to ensure consistency
+ across implementations.
+
+ Fixed values by type:
+ - boolean: false
+ - int: 0
+ - long: 0
+ - float: 0.0
+ - double: 0.0
+ - decimal(p, s): 0 (the unscaled value is 0)
+ - string: "XXXXXXXX"
+ - date: 1970-01-01
+ - time: 00:00:00
+ - timestamp: 1970-01-01T00:00:00
+ - timestamptz: 1970-01-01T00:00:00+00:00
+ - timestamp_ns: 1970-01-01T00:00:00.000000000
+ - timestamptz_ns: 1970-01-01T00:00:00.000000000+00:00
+ - uuid: 00000000-0000-0000-0000-000000000000
+ - fixed(n): n zero bytes
+ - binary: empty byte sequence
+ - variant: {}
+ - geometry: POINT EMPTY
+ - geography: POINT EMPTY
+ - list: empty list []
+ - map: empty map {}
+ - struct: struct with each field set to its type-specific default
(applied recursively)
+
+ NULL input is also replaced with the type-specific fixed value; NULL
is not preserved.
+
+ Applicable to: all data types
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-to-fixed-value"
+
+ ReplaceWithNull:
+ description: >
+ Replaces the entire column value with NULL. NULL input is preserved
(NULL -> NULL).
+ A server must not return this action for a required (non-nullable)
column.
+
+ Applicable to: all optional types
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "replace-with-null"
+
+ ShowFirst4:
+ description: >
+ Preserves the first 4 Unicode code points of the column value and
redacts the remainder
+ using mask-alphanum rules (see MaskAlphanum for the exact character
rules).
+ Values with 4 or fewer Unicode code points are returned unchanged.
+
+ For example: "[email protected]" -> "[email protected]"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "show-first-4"
+
+ ShowLast4:
+ description: >
+ Redacts all Unicode code points except the last 4 using mask-alphanum
rules
+ (see MaskAlphanum for the exact character rules).
+ Values with 4 or fewer Unicode code points are returned unchanged.
+
+ For example: "4111-1111-1111-4444" -> "nnnn-nnnn-nnnn-4444"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "show-last-4"
+
+ TruncateToYear:
+ description: >
+ Truncates the column value to year precision, setting month, day, and
time components
+ to their minimum values. The output type matches the input type.
+
+ For example: 2024-07-15 -> 2024-01-01
+ For timestamptz and timestamptz_ns, truncation is performed in UTC.
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: date, timestamp, timestamptz, timestamp_ns,
timestamptz_ns
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "truncate-to-year"
+
+ TruncateToMonth:
+ description: >
+ Truncates the column value to year and month precision, setting day
and time components
+ to their minimum values. The output type matches the input type.
+
+ For example: 2024-07-15 -> 2024-07-01
+ For timestamptz and timestamptz_ns, truncation is performed in UTC.
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: date, timestamp, timestamptz, timestamp_ns,
timestamptz_ns
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "truncate-to-month"
+
+ Sha256Global:
+ description: |
+ Applies SHA-256. Deterministic across all queries
+ and engines: the same input always produces the same output.
Review Comment:
nit - we use the term `readers` in describing other masking actions, but we
use `engines` here:
```suggestion
and readers: the same input always produces the same output.
```
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -3632,6 +3641,301 @@ components:
additionalProperties:
type: string
+ ReadRestrictions:
+ type: object
+ description: >
+ Read restrictions for a table.
+
+ A reader evaluates the row filter against original, untransformed
column
+ values, then applies required-column-projections to the surviving
rows.
+ Each action must produce a value of the same type as the input
column.
+ If a reader cannot apply any returned restriction (a filter
expression
+ or an action), it must fail the query and must not silently return
raw,
+ partial, or empty results.
+
+ If a projection targets a field, that action governs every value
reachable
+ through it; other projections in the same object that target a
descendant
+ of that field have no effect.
+
+ An empty ReadRestrictions object (no required-column-projections and
no
+ required-row-filter) imposes no restrictions and is equivalent to the
+ field being absent from the response.
+ example:
+ required-column-projections:
+ - field-id: 4
+ action: show-last-4
+ - field-id: 6
+ action: replace-with-null
+ - field-id: 8
+ action: truncate-to-year
+ - field-id: 10
+ action: sha-256-global
+ - field-id: 12
+ action: mask-alphanum
+ required-row-filter:
+ type: eq
+ term: region
+ value: US
+ properties:
+ required-column-projections:
+ description: >
+ A list of columns that require specific actions to be applied when
reading.
+ A server must not return an action for a column whose type is not
listed in
+ that action's "Applicable to" set. If absent or empty, no required
actions
+ apply; columns not listed are not subject to any required action.
+
+ 1. For each column listed, the reader must apply the specified
action before
+ returning values for that column.
+
+ 2. The reader must replace all output references to the column
with the result
+ of the action, presenting the result under the original column
name. For
+ example, if the action for column 'cc' is mask-alphanum, the
reader must
+ return the masked value as 'cc' in the query output.
+
+ 3. A column must appear at most once in
required-column-projections.
+ type: array
+ items:
+ $ref: '#/components/schemas/Action'
+ required-row-filter:
+ description: >
+ An expression that filters rows in the table that the
authenticated principal does not have access to.
+
+ 1. The expression must evaluate to a boolean (TRUE or FALSE;
Iceberg expressions
+ never produce NULL). A reader must discard any row for which the
filter
+ evaluates to FALSE, and no information derived from discarded
rows may be
+ included in the query result.
+
+ 2. If this property is absent, null, or always true then no
mandatory filtering is required.
+ allOf:
+ - $ref: '#/components/schemas/Expression'
+
+ Action:
+ discriminator:
+ propertyName: action
+ mapping:
+ mask-alphanum: '#/components/schemas/MaskAlphanum'
+ mask-to-fixed-value: '#/components/schemas/MaskToFixedValue'
+ replace-with-null: '#/components/schemas/ReplaceWithNull'
+ show-first-4: '#/components/schemas/ShowFirst4'
+ show-last-4: '#/components/schemas/ShowLast4'
+ truncate-to-year: '#/components/schemas/TruncateToYear'
+ truncate-to-month: '#/components/schemas/TruncateToMonth'
+ sha-256-global: '#/components/schemas/Sha256Global'
+ sha-256-query-local: '#/components/schemas/Sha256QueryLocal'
+ type: object
+ required:
+ - action
+ - field-id
+ properties:
+ action:
+ type: string
+ field-id:
+ type: integer
+ description: Field ID of the column being projected.
+
+ MaskAlphanum:
+ description: >
+ Redacts the column value using the following rules to transform
Unicode code points:
+
+ - Digits (U+0030–U+0039, 0-9) are replaced with 'n'
+ - The following punctuation characters are kept as-is:
+ U+0028 '(' LEFT PARENTHESIS
+ U+0029 ')' RIGHT PARENTHESIS
+ U+002C ',' COMMA
+ U+002E '.' FULL STOP
+ U+002D '-' HYPHEN-MINUS
+ U+0040 '@' COMMERCIAL AT
+ - All other Unicode characters (including letters, whitespace, and any
punctuation
+ not listed above) are replaced with 'x'
+
+ For example: "[email protected]" -> "[email protected]"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-alphanum"
+
+ MaskToFixedValue:
+ description: >
+ Replaces the column value with a type-specific fixed value.
+ Readers must use exactly the values listed below to ensure consistency
+ across implementations.
+
+ Fixed values by type:
+ - boolean: false
+ - int: 0
+ - long: 0
+ - float: 0.0
+ - double: 0.0
+ - decimal(p, s): 0 (the unscaled value is 0)
+ - string: "XXXXXXXX"
+ - date: 1970-01-01
+ - time: 00:00:00
+ - timestamp: 1970-01-01T00:00:00
+ - timestamptz: 1970-01-01T00:00:00+00:00
+ - timestamp_ns: 1970-01-01T00:00:00.000000000
+ - timestamptz_ns: 1970-01-01T00:00:00.000000000+00:00
+ - uuid: 00000000-0000-0000-0000-000000000000
+ - fixed(n): n zero bytes
+ - binary: empty byte sequence
+ - variant: {}
+ - geometry: POINT EMPTY
+ - geography: POINT EMPTY
+ - list: empty list []
+ - map: empty map {}
+ - struct: struct with each field set to its type-specific default
(applied recursively)
+
+ NULL input is also replaced with the type-specific fixed value; NULL
is not preserved.
+
+ Applicable to: all data types
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "mask-to-fixed-value"
+
+ ReplaceWithNull:
+ description: >
+ Replaces the entire column value with NULL. NULL input is preserved
(NULL -> NULL).
+ A server must not return this action for a required (non-nullable)
column.
+
+ Applicable to: all optional types
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "replace-with-null"
+
+ ShowFirst4:
+ description: >
+ Preserves the first 4 Unicode code points of the column value and
redacts the remainder
+ using mask-alphanum rules (see MaskAlphanum for the exact character
rules).
+ Values with 4 or fewer Unicode code points are returned unchanged.
+
+ For example: "[email protected]" -> "[email protected]"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "show-first-4"
+
+ ShowLast4:
+ description: >
+ Redacts all Unicode code points except the last 4 using mask-alphanum
rules
+ (see MaskAlphanum for the exact character rules).
+ Values with 4 or fewer Unicode code points are returned unchanged.
+
+ For example: "4111-1111-1111-4444" -> "nnnn-nnnn-nnnn-4444"
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "show-last-4"
+
+ TruncateToYear:
+ description: >
+ Truncates the column value to year precision, setting month, day, and
time components
+ to their minimum values. The output type matches the input type.
+
+ For example: 2024-07-15 -> 2024-01-01
+ For timestamptz and timestamptz_ns, truncation is performed in UTC.
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: date, timestamp, timestamptz, timestamp_ns,
timestamptz_ns
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "truncate-to-year"
+
+ TruncateToMonth:
+ description: >
+ Truncates the column value to year and month precision, setting day
and time components
+ to their minimum values. The output type matches the input type.
+
+ For example: 2024-07-15 -> 2024-07-01
+ For timestamptz and timestamptz_ns, truncation is performed in UTC.
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: date, timestamp, timestamptz, timestamp_ns,
timestamptz_ns
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "truncate-to-month"
+
+ Sha256Global:
+ description: |
+ Applies SHA-256. Deterministic across all queries
+ and engines: the same input always produces the same output.
+
+ Input-to-bytes encoding by type:
+ - string: UTF-8 encoded bytes
+ - int: 4 bytes, little-endian
+ - long: 8 bytes, little-endian
+ - binary: raw bytes as-is
+
+ Output encoding by type:
+ - string: 64-character lowercase hexadecimal string
+ - int: first 4 bytes of the digest, read as a little-endian int
+ - long: first 8 bytes of the digest, read as a little-endian long
+ - binary: the full 32-byte raw SHA-256 digest
+
+ NULL input is preserved (NULL -> NULL).
+
+ Applicable to: string, int, long, binary
+ allOf:
+ - $ref: '#/components/schemas/Action'
+ properties:
+ action:
+ type: string
+ const: "sha-256-global"
+
+ Sha256QueryLocal:
+ description: |
+ Applies SHA-256 with a per-query random salt, making the output
non-deterministic
+ across queries while remaining consistent within a single query. The
definition
+ of a query is left to the implementation.
+
+ The engine must generate a cryptographically random salt of at least
16 bytes for each query.
+
+ For each column value, the engine must encode the value to bytes using
+ sha-256-global's input rules, prepend the per-query salt, and compute
Review Comment:
nit, similar as above:
```suggestion
The reader must generate a cryptographically random salt of at least
16 bytes for each query.
For each column value, the reader must encode the value to bytes
using
sha-256-global's input rules, prepend the per-query salt, and compute
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
