bennychow commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r4050453756
########## format/view-spec.md: ########## @@ -160,7 +178,121 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +#### Freshness + +A materialized view is "fresh" when the storage table adequately represents the result of the view query at the current state of its dependencies. +Since different systems define freshness differently, it is left to the consumer to evaluate freshness based on its own policy. + +**Consumer behavior:** + +When evaluating freshness, consumers: + +- May apply time-based freshness policies, such as allowing a staleness window based on `refresh-start-timestamp-ms`. +- May compare the `source-states` list against the states loaded from the catalog to verify the producer's freshness interpretation. +- May parse the view definition to implement more sophisticated policies. +- When a materialized view is considered stale, can fail, refresh inline, or treat the materialized view as a logical view. Review Comment: Missing subject. **consumer** can fail. ########## format/view-spec.md: ########## @@ -322,3 +454,96 @@ s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json } ] } ``` + +### Materialized View Example + +Imagine the following operation, which creates a materialized view that precomputes daily event counts: + +```sql +USE prod.default +``` +```sql +CREATE MATERIALIZED VIEW event_agg_mv ( + event_count COMMENT 'Count of events', + event_date) +COMMENT 'Precomputed daily event counts' +AS +SELECT + COUNT(1), CAST(event_ts AS DATE) +FROM events +GROUP BY 2 +``` + +The materialized view metadata JSON file looks as follows: + +``` +s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json +``` +```json +{ + "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35", + "format-version" : 1, + "location" : "s3://bucket/warehouse/default.db/event_agg_mv", + "current-version-id" : 1, + "properties" : { + "comment" : "Precomputed daily event counts" + }, + "versions" : [ { + "version-id" : 1, + "timestamp-ms" : 1573518431292, + "schema-id" : 1, + "default-catalog" : "prod", + "default-namespace" : [ "default" ], + "summary" : { + "engine-name" : "Spark", + "engine-version" : "3.4.1" + }, + "representations" : [ { + "type" : "sql", + "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2", + "dialect" : "spark" + } ], + "storage-table" : { + "namespace" : [ "default" ], + "name" : "event_agg_mv__storage" + } + } ], + "schemas": [ { + "schema-id": 1, + "type" : "struct", + "fields" : [ { + "id" : 1, + "name" : "event_count", + "required" : false, + "type" : "int", + "doc" : "Count of events" + }, { + "id" : 2, + "name" : "event_date", + "required" : false, + "type" : "date" + } ] + } ], + "version-log" : [ { + "timestamp-ms" : 1573518431292, + "version-id" : 1 + } ] +} +``` + +After a refresh operation, the storage table's snapshot summary contains the `refresh-state` property. +The following is an example of the `refresh-state` JSON value stored in the snapshot summary of the storage table: + +```json +{ + "view-version-id" : 1, + "refresh-start-timestamp-ms" : 1573518435000, + "source-states" : [ { + "type" : "table", + "namespace" : [ "default" ], Review Comment: Since catalog will be required because we only want to capture fully resolved identifiers, we should add "catalog: prod" here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
