stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4589267908


##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.
+
+#### Refresh state
+
+Query engines maintain the precomputed data in the storage table by executing 
the materialized view’s query. During a refresh, the engine records the state 
of the view’s dependencies in the refresh state. Which dependencies are 
available depends on how source materialized views are treated, and the writer 
chooses which of them to record: all, a subset, or none. See [Appendix 
B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for 
examples. Each recorded dependency appears in `source-states` as a table entry 
or a view entry. A source materialized view may be recorded as a view entry for 
its definition and, optionally, a table entry for its storage table.
+
+The refresh state has the following fields:
+
+| Requirement | Field name                   | Description |
+|-------------|------------------------------|-------------|
+| _required_  | `refresh-start-timestamp-ms` | A timestamp of when the refresh 
operation was started |
+| _required_  | `view-version-id`            | The `version-id` of the 
materialized view when the refresh operation was performed |
+| _required_  | `source-states`              | A list of [source 
state](#source-state) records capturing the dependencies the writer chose to 
record; may be empty |
+
+When writing the refresh state, writers may leave source-states empty (e.g., 
when sources are non-Iceberg tables or a mechanism outside this spec determines 
whether the storage table reflects the result of the view query).
+
+##### Source state
+
+Source state records capture the state of objects referenced by a materialized 
view. Each record has a `type` field that determines its form:
+
+| Type    | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, 
or the storage table of a source materialized view |
+| `view`  | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its 
view metadata and one or more `table` entries referencing its storage table or 
other source tables. These source table entries might be determined by 
recursively expanding its own dependencies.

Review Comment:
   > A source materialized view may be recorded as a `view` entry referencing 
its view metadata and one or more `table` entries referencing its storage table 
or other source tables.
   
   We seem to combine two things here: how a source MV itself is recorded, and 
the recursive expansion of its dependencies. I know `other source tables` is to 
allow recursive expansion, but the current wording would exclude nested view 
entities. "One or more `table` entries" also implies the storage table entry is 
required, while the Refresh state section says it is optional, and the Appendix 
B example records `event_agg_mv` as a view entry only.
   
   The Refresh state section already describes how a source MV is recorded, and 
the type table says "in the dependency graph", which includes nested 
dependencies. So this sentence could be removed. If we want to call out 
expansion explicitly, maybe:
   
   > A source materialized view may be recorded as a `view` entry for its 
definition and, optionally, a `table` entry for its storage table. If a writer 
expands a source materialized view's own dependencies, each expanded dependency 
is recorded as a `table` or `view` entry like any other dependency.
   



##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.
+
+#### Refresh state
+
+Query engines maintain the precomputed data in the storage table by executing 
the materialized view’s query. During a refresh, the engine records the state 
of the view’s dependencies in the refresh state. Which dependencies are 
available depends on how source materialized views are treated, and the writer 
chooses which of them to record: all, a subset, or none. See [Appendix 
B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for 
examples. Each recorded dependency appears in `source-states` as a table entry 
or a view entry. A source materialized view may be recorded as a view entry for 
its definition and, optionally, a table entry for its storage table.
+
+The refresh state has the following fields:
+
+| Requirement | Field name                   | Description |
+|-------------|------------------------------|-------------|
+| _required_  | `refresh-start-timestamp-ms` | A timestamp of when the refresh 
operation was started |
+| _required_  | `view-version-id`            | The `version-id` of the 
materialized view when the refresh operation was performed |
+| _required_  | `source-states`              | A list of [source 
state](#source-state) records capturing the dependencies the writer chose to 
record; may be empty |
+
+When writing the refresh state, writers may leave source-states empty (e.g., 
when sources are non-Iceberg tables or a mechanism outside this spec determines 
whether the storage table reflects the result of the view query).
+
+##### Source state
+
+Source state records capture the state of objects referenced by a materialized 
view. Each record has a `type` field that determines its form:
+
+| Type    | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, 
or the storage table of a source materialized view |
+| `view`  | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its 
view metadata and one or more `table` entries referencing its storage table or 
other source tables. These source table entries might be determined by 
recursively expanding its own dependencies.
+
+##### Source table state
+
+A source table record captures the state of a source table (including a source 
materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name    | Description |
+|-------------|---------------|-------------|
+| _required_  | `type`        | A string that must be set to `table` |
+| _required_  | `name`        | A string specifying the name of the source 
table |
+| _required_  | `namespace`   | A list of strings for namespace levels |
+| _required_  | `catalog`     | A string specifying the name of the catalog. |
+| _required_  | `uuid`        | The uuid of the source table |
+| _optional_  | `snapshot-id` | The snapshot-id of the source table that was 
read during the refresh operation. Omitted when the source table had no 
snapshots at refresh time (e.g. a newly created, empty table) |
+| _optional_  | `ref`         | Branch or tag name of the source table being 
referenced in the view query |
+
+When `ref` is set, the source table is tracked against that branch or tag. 
When `ref` is not set, the source table is pinned to the recorded `snapshot-id` 
(e.g. `FOR VERSION AS OF <snapshot-id>`). Writers that read a branch or tag 
should record its name in `ref`.
+
+##### Source view state
+
+A source view record captures the state of a source view at the time of the 
last refresh operation.
+
+| Requirement | Field name   | Description |
+|-------------|--------------|-------------|
+| _required_  | `type`       | A string that must be set to `view` |
+| _required_  | `name`       | A string specifying the name of the source view 
|
+| _required_  | `namespace`  | A list of strings for namespace levels |
+| _required_  | `catalog`     | A string specifying the name of the catalog. |
+| _required_  | `uuid`       | The uuid of the source view |
+| _required_  | `version-id` | The version-id of the source view that was read 
during the refresh operation |
+
+#### Storage table creation and configuration
+
+When processing a `CREATE MATERIALIZED VIEW` statement, query engines should:
+
+1. Create the storage table as a regular Iceberg table with any specified 
configurations (partitioning, sort order, compression, etc.).

Review Comment:
   nit: drop the word `any`?



##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another 
provides the basis for maki
 
 Writers create view metadata files optimistically, assuming that the current 
metadata location will not be changed before the writer's commit. Once a writer 
has created an update, it commits by swapping the view's metadata file pointer 
from the base location to the new location.
 
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view 
query stored as a table.
+When queried, engines may return the precomputed data for the materialized 
views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view 
and an underlying Iceberg table, the "storage-table", which stores the 
precomputed data.
+Materialized View metadata is a superset of View metadata with an additional 
pointer to the storage table. The storage table is an Iceberg table with 
additional materialized view refresh state metadata.
+This addition is backward compatible: a reader that is unaware of materialized 
views sees the unknown `storage-table` field, ignores it, and evaluates the 
view query directly.
+Refresh metadata contains information about the "source tables", "source 
views", and/or "source materialized views", which are the 
tables/views/materialized views used in the computation of the query results of 
the materialized view.

Review Comment:
   nit: `and/or` -> `and`



##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.

Review Comment:
   After storage table maintenance (e.g., a compaction that commits a `replace` 
snapshot), the current snapshot has no `refresh-state`. Should readers use the 
most recent snapshot with `refresh-state` in the ancestry of the storage 
table's `main` branch? And does a later snapshot that changes data (anything 
other than `replace`) invalidate that state?
   
   What do you think of sth below? is the 2nd sentence over specified?
   
   > The refresh state of a storage table is the `refresh-state` of the most 
recent snapshot in the history of its `main` branch that has one. If a later 
snapshot in that history changes the table's data (any operation other than 
`replace`), the storage table no longer reflects that refresh state.



##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.
+
+#### Refresh state
+
+Query engines maintain the precomputed data in the storage table by executing 
the materialized view’s query. During a refresh, the engine records the state 
of the view’s dependencies in the refresh state. Which dependencies are 
available depends on how source materialized views are treated, and the writer 
chooses which of them to record: all, a subset, or none. See [Appendix 
B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for 
examples. Each recorded dependency appears in `source-states` as a table entry 
or a view entry. A source materialized view may be recorded as a view entry for 
its definition and, optionally, a table entry for its storage table.
+
+The refresh state has the following fields:
+
+| Requirement | Field name                   | Description |
+|-------------|------------------------------|-------------|
+| _required_  | `refresh-start-timestamp-ms` | A timestamp of when the refresh 
operation was started |

Review Comment:
   nit: add the unit, matching `timestamp-ms` elsewhere in the spec.
   
   ```suggestion
   | _required_  | `refresh-start-timestamp-ms` | Timestamp when the refresh 
operation was started (ms from epoch) |
   ```
   



##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.
+
+#### Refresh state
+
+Query engines maintain the precomputed data in the storage table by executing 
the materialized view’s query. During a refresh, the engine records the state 
of the view’s dependencies in the refresh state. Which dependencies are 
available depends on how source materialized views are treated, and the writer 
chooses which of them to record: all, a subset, or none. See [Appendix 
B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for 
examples. Each recorded dependency appears in `source-states` as a table entry 
or a view entry. A source materialized view may be recorded as a view entry for 
its definition and, optionally, a table entry for its storage table.
+
+The refresh state has the following fields:
+
+| Requirement | Field name                   | Description |
+|-------------|------------------------------|-------------|
+| _required_  | `refresh-start-timestamp-ms` | A timestamp of when the refresh 
operation was started |
+| _required_  | `view-version-id`            | The `version-id` of the 
materialized view when the refresh operation was performed |
+| _required_  | `source-states`              | A list of [source 
state](#source-state) records capturing the dependencies the writer chose to 
record; may be empty |
+
+When writing the refresh state, writers may leave source-states empty (e.g., 
when sources are non-Iceberg tables or a mechanism outside this spec determines 
whether the storage table reflects the result of the view query).

Review Comment:
   An empty or partial `source-states` looks the same whether the writer's 
criteria didn't need sources or the writer couldn't record them. Freshness is 
intentionally left to engines. Maybe we can update this paragraphs so that it 
states what readers can't infer from the refresh state.
   
   ```suggestion
   `source-states` may be empty or cover only some of the view query's 
dependencies (e.g., when sources are non-Iceberg tables or a mechanism outside 
this spec determines whether the storage table reflects the result of the view 
query). A dependency that is not recorded carries no information about its 
state.
   ```
   



##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.
+
+#### Refresh state
+
+Query engines maintain the precomputed data in the storage table by executing 
the materialized view’s query. During a refresh, the engine records the state 
of the view’s dependencies in the refresh state. Which dependencies are 
available depends on how source materialized views are treated, and the writer 
chooses which of them to record: all, a subset, or none. See [Appendix 
B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for 
examples. Each recorded dependency appears in `source-states` as a table entry 
or a view entry. A source materialized view may be recorded as a view entry for 
its definition and, optionally, a table entry for its storage table.
+
+The refresh state has the following fields:
+
+| Requirement | Field name                   | Description |
+|-------------|------------------------------|-------------|
+| _required_  | `refresh-start-timestamp-ms` | A timestamp of when the refresh 
operation was started |
+| _required_  | `view-version-id`            | The `version-id` of the 
materialized view when the refresh operation was performed |
+| _required_  | `source-states`              | A list of [source 
state](#source-state) records capturing the dependencies the writer chose to 
record; may be empty |
+
+When writing the refresh state, writers may leave source-states empty (e.g., 
when sources are non-Iceberg tables or a mechanism outside this spec determines 
whether the storage table reflects the result of the view query).
+
+##### Source state
+
+Source state records capture the state of objects referenced by a materialized 
view. Each record has a `type` field that determines its form:
+
+| Type    | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, 
or the storage table of a source materialized view |
+| `view`  | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its 
view metadata and one or more `table` entries referencing its storage table or 
other source tables. These source table entries might be determined by 
recursively expanding its own dependencies.
+
+##### Source table state
+
+A source table record captures the state of a source table (including a source 
materialized view's storage table) at the time of the last refresh operation.
+
+| Requirement | Field name    | Description |
+|-------------|---------------|-------------|
+| _required_  | `type`        | A string that must be set to `table` |
+| _required_  | `name`        | A string specifying the name of the source 
table |
+| _required_  | `namespace`   | A list of strings for namespace levels |
+| _required_  | `catalog`     | A string specifying the name of the catalog. |
+| _required_  | `uuid`        | The uuid of the source table |
+| _optional_  | `snapshot-id` | The snapshot-id of the source table that was 
read during the refresh operation. Omitted when the source table had no 
snapshots at refresh time (e.g. a newly created, empty table) |
+| _optional_  | `ref`         | Branch or tag name of the source table being 
referenced in the view query |
+
+When `ref` is set, the source table is tracked against that branch or tag. 
When `ref` is not set, the source table is pinned to the recorded `snapshot-id` 
(e.g. `FOR VERSION AS OF <snapshot-id>`). Writers that read a branch or tag 
should record its name in `ref`.

Review Comment:
   If the view query reads the source table without a branch or time-travel 
clause, it reads `main` implicitly. Should the writer record `ref: main` in 
that case? Otherwise the entry is treated as pinned to `snapshot-id`, and later 
commits to the source table would never show up as changes. Since omitting 
`ref` changes the meaning this much, I am wondering if this should be a `must`.
   
   Also, if `ref` is not set and `snapshot-id` is omitted (empty source table), 
what is the entry pinned to?
   
   Maybe sth like:
   
   ```suggestion
   When `ref` is set, the source table is tracked against that branch or tag. 
When `ref` is not set, the source table is pinned to the recorded `snapshot-id` 
(e.g. `FOR VERSION AS OF <snapshot-id>`). Writers must record the branch or tag 
name in `ref` when the source table is read by branch or tag, including the 
implicit `main` branch when the query does not specify one.
   ```
   



##########
format/view-spec.md:
##########
@@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following 
fields:
 | _required_  | `representations`   | A list of 
[representations](#representations) for the view definition         |
 | _optional_  | `default-catalog`   | Catalog name to use when a reference in 
the SELECT does not contain a catalog |
 | _required_  | `default-namespace` | Namespace to use when a reference in the 
SELECT is a single identifier        |
+| _optional_  | `storage-table`     | A [storage table 
identifier](#storage-table-identifier) of the storage table |

Review Comment:
   Just to close the loop. 
   
   @bennychow laid out the case for per-version storage tables in 
https://github.com/apache/iceberg/pull/11041#discussion_r2809831576: a view can 
gain, drop, or swap its storage table across versions, and a full refresh can 
write a new storage table and commit it with a new view version instead of 
truncating the old one.
   
   The Feb 19 sync noted that no engine we know of supports DDL to convert a 
logical view to an MV. So the full-refresh case seems to be the main argument 
for keeping `storage-table` in `Version`. I am fine with keeping the reference 
inside the version struct.
   



##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.
+
+#### Refresh state
+
+Query engines maintain the precomputed data in the storage table by executing 
the materialized view’s query. During a refresh, the engine records the state 
of the view’s dependencies in the refresh state. Which dependencies are 
available depends on how source materialized views are treated, and the writer 
chooses which of them to record: all, a subset, or none. See [Appendix 
B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for 
examples. Each recorded dependency appears in `source-states` as a table entry 
or a view entry. A source materialized view may be recorded as a view entry for 
its definition and, optionally, a table entry for its storage table.
+
+The refresh state has the following fields:
+
+| Requirement | Field name                   | Description |
+|-------------|------------------------------|-------------|
+| _required_  | `refresh-start-timestamp-ms` | A timestamp of when the refresh 
operation was started |
+| _required_  | `view-version-id`            | The `version-id` of the 
materialized view when the refresh operation was performed |
+| _required_  | `source-states`              | A list of [source 
state](#source-state) records capturing the dependencies the writer chose to 
record; may be empty |
+
+When writing the refresh state, writers may leave source-states empty (e.g., 
when sources are non-Iceberg tables or a mechanism outside this spec determines 
whether the storage table reflects the result of the view query).
+
+##### Source state
+
+Source state records capture the state of objects referenced by a materialized 
view. Each record has a `type` field that determines its form:
+
+| Type    | Description |
+|---------|-------------|
+| `table` | An Iceberg table — either a source table in the dependency graph, 
or the storage table of a source materialized view |
+| `view`  | An Iceberg view in the dependency graph |
+
+A source materialized view may be recorded as a `view` entry referencing its 
view metadata and one or more `table` entries referencing its storage table or 
other source tables. These source table entries might be determined by 
recursively expanding its own dependencies.
+
+##### Source table state
+
+A source table record captures the state of a source table (including a source 
materialized view's storage table) at the time of the last refresh operation.

Review Comment:
   nit: "the last refresh operation" reads as the most recent refresh, but a 
reader may be looking at an older storage table snapshot. Same for the source 
view state description.
   
   ```suggestion
   A source table record captures the state of a source table (including a 
source materialized view's storage table) at the time of the refresh operation 
that wrote this refresh state.
   ```
   



##########
format/view-spec.md:
##########
@@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following 
fields:
 | _required_  | `timestamp-ms` | Timestamp when the view's 
`current-version-id` was updated (ms from epoch) |
 | _required_  | `version-id`   | ID that `current-version-id` was set to |
 
-## Appendix A: An Example
+#### Storage Table Identifier
+
+The table identifier for the storage table that stores the precomputed results.
+
+| Requirement | Field name     | Description |
+|-------------|----------------|-------------|
+| _required_  | `namespace`    | A list of strings for namespace levels |
+| _required_  | `name`         | A string specifying the name of the table |
+| _optional_  | `catalog`      | A string specifying the name of the catalog. 
When `null` or not set, the catalog in which the materialized view is stored is 
used |
+
+### Storage table metadata
+
+This section describes additional metadata for the storage table that 
supplements the regular table metadata and is required for materialized views.
+The `refresh-state` property is set on the [snapshot 
summary](https://iceberg.apache.org/spec/#snapshots) property of a storage 
table snapshot to provide information about the state of the precomputed data.
+
+| Requirement | Field name      | Description |
+|-------------|-----------------|-------------|
+| _optional_  | `refresh-state` | A [refresh state](#refresh-state) record 
stored as a JSON-encoded string |
+
+Snapshots written by a refresh operation must set `refresh-state`.
+
+#### Refresh state
+
+Query engines maintain the precomputed data in the storage table by executing 
the materialized view’s query. During a refresh, the engine records the state 
of the view’s dependencies in the refresh state. Which dependencies are 
available depends on how source materialized views are treated, and the writer 
chooses which of them to record: all, a subset, or none. See [Appendix 
B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for 
examples. Each recorded dependency appears in `source-states` as a table entry 
or a view entry. A source materialized view may be recorded as a view entry for 
its definition and, optionally, a table entry for its storage table.

Review Comment:
   nit: "Which dependencies are available depends on how source materialized 
views are treated" is hard to act on: it's unclear what "available" means here. 
The rest of the sentence already says the writer chooses what to record. Could 
we drop that clause?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to