tomtongue opened a new pull request, #16859:
URL: https://github.com/apache/iceberg/pull/16859
## Overview
This PR adds a read-only `table_properties_log` metadata table that exposes
the history of a table's effective properties from Iceberg metadata files.
For each retained metadata version, the table returns the metadata file
timestamp, the metadata file location, the snapshot that was current in that
metadata version, and the full `properties` map recorded in that
`metadata.json`.
This makes table-property history queryable from the table's own metadata.
It allows users to answer questions such as which properties were in effect
when a snapshot became current, when a property change appeared in table
metadata, and whether a later incident or behavior change lines up with a
table-property update.
The change is read-only and engine-level. It does not change the table spec,
does not require a format-version bump, and does not change the write path. The
new table is registered through the existing metadata-table mechanism, so it is
available to Spark and Flink using their existing metadata-table syntax.
## Motivation
Iceberg versions table state through immutable metadata files. Schemas,
partition specs, sort orders, snapshots, `snapshot-log` entries, and
`metadata-log` entries are retained as part of table metadata. However,
`TableMetadata.properties` is represented as the current effective property map
in each metadata file. When a property is changed, the current metadata points
at the new map, and there is no direct query surface for comparing that map
across previous metadata versions.
* The historical values are still present as long as the corresponding
metadata files are retained. Each historical `metadata.json` contains the full
property map for that metadata version, and `TableMetadata.previousFiles()`
records the retained previous metadata files. Using that information requires
locating and parsing those files manually today.
* That gap matters for operational debugging and audit-style questions. For
example, an operator may need to determine whether `gc.enabled` was enabled
before an expiration run, whether a performance regression started after
`write.target-file-size-bytes` or `write.distribution-mode` changed, or whether
a read behavior change lines up with `write.delete.mode`, `write.update.mode`,
or `write.merge.mode` and more.
`table_properties_log` exposes the history Iceberg already retains, without
adding new persisted metadata and without requiring out-of-band catalog logs.
## Use-cases
The primary use-cases are point-in-time inspection of table configuration
from Iceberg metadata.
* **Audit and forensic checks.** Query whether a property had a specific
value in a retained metadata version, including values that were later changed
back.
* **Incident RCA.** Correlate a behavior change with the metadata version
where a property first changed.
* **Performance tuning.** Verify that tuning properties took effect and
compare the values used before and after a change.
* **Engine-agnostic access.** Use the same core metadata table from Spark
and Flink through the existing metadata-table resolution path.
## Changes
This PR makes the following changes:
* Adds `TablePropertiesLogTable` in core.
* The table is modeled on `MetadataLogEntriesTable` and the existing
static metadata-table pattern.
* It reads `TableMetadata.previousFiles()` and appends the current
metadata file as the latest row.
* Each row is built by reading the corresponding `metadata.json` with
`TableMetadataParser.read`.
* Adds `MetadataTableType.TABLE_PROPERTIES_LOG`.
* The metadata table can be resolved by name through
`MetadataTableType.from`.
* Registers `TABLE_PROPERTIES_LOG` in `MetadataTableUtils`.
* Spark can query it as `<table>.table_properties_log`.
* Flink can query it as `<table>$table_properties_log`.
* No Spark-specific or Flink-specific implementation is needed.
* Exposes the following schema:
| Column | Type | Description |
| --- | --- | --- |
| `timestamp` | `timestamp with zone` | Timestamp of the metadata log
entry |
| `file` | `string` | Location of the `metadata.json` file |
| `latest_snapshot_id` | `long` | Snapshot that was current in that
metadata version; `null` before the first snapshot |
| `properties` | `map<string, string>` | Effective table properties stored
in that metadata version |
* Add the tests verifying the metadata table query results to Spark 4.1 and
Flink 2.1
Note that:
* The visible history is bounded by retained metadata files. In practice
this is controlled by metadata retention settings such as
`write.metadata.previous-versions-max` and whether old metadata files are
deleted after commit.
* The table reads one retained `metadata.json` per emitted metadata version.
This is bounded by the metadata log exposed by the current metadata, not by
every metadata file ever written for the table.
* Predicate filtering is handled by the metadata-table scan path and
engines; this PR does not add predicate pushdown into historical metadata-file
reads.
* `latest_snapshot_id` is the snapshot current in that metadata version. A
metadata-only property update can therefore produce a row with the same
`latest_snapshot_id` as the previous row.
### Example: Query on TablePropertiesLogTable with Spark
```sql
CREATE TABLE props_log (id int, name string) USING iceberg TBLPROPERTIES
('key1'='value1')";
SELECT * FROM props_log.table_properties_log;
/*
+-----------------------+---------------------------------------------------------------------------------------------+------------------+-------------------------------------------------------------------------+
|timestamp |file
|latest_snapshot_id|properties
|
+-----------------------+---------------------------------------------------------------------------------------------+------------------+-------------------------------------------------------------------------+
|2026-06-17
15:24:33.798|s3://warehouse/db/props_log/metadata/00000-059abc2d-e741-4465-ae11-121b8023d380.metadata.json|NULL
|{key1 -> value1, owner -> spark, write.parquet.compression-codec
-> zstd}|
+-----------------------+---------------------------------------------------------------------------------------------+------------------+-------------------------------------------------------------------------+
*/
ALTER TABLE db.props_log SET TBLPROPERTIES('key2'='value2');
INSERT INTO db.props_log VALUES (1, 'a');
/*
+-----------------------+---------------------------------------------------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+
|timestamp |file
|latest_snapshot_id |properties
|
+-----------------------+---------------------------------------------------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+
|2026-06-17
15:24:33.798|s3://warehouse/db/props_log/metadata/00000-059abc2d-e741-4465-ae11-121b8023d380.metadata.json|NULL
|{key1 -> value1, owner -> spark,
write.parquet.compression-codec -> zstd} |
|2026-06-17
15:25:17.637|s3://warehouse/db/props_log/metadata/00001-28f3023b-423f-4b4c-bc18-ef93596baffe.metadata.json|NULL
|{key1 -> value1, owner -> spark, key2 -> value2,
write.parquet.compression-codec -> zstd}|
|2026-06-17
15:26:09.574|s3://warehouse/db/props_log/metadata/00002-6b48f0eb-a243-4146-9890-0400deba828b.metadata.json|1312307084484665830|{key1
-> value1, owner -> spark, key2 -> value2, write.parquet.compression-codec ->
zstd}|
+-----------------------+---------------------------------------------------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+
*/
```
## Alternatives Considered
* **Record table properties into each snapshot summary at commit time.**:
This was the original design direction, and the full design is captured in
[Preserving Table Property History in Snapshot
Summary](https://docs.google.com/document/d/1w3dQ_-m6rqSpsaLGuXTSUYVBNQn3UiYBCWEgt9ztTJo/edit?tab=t.0).
That approach would make property history available from `<table>.snapshots`
and tie values directly to snapshot retention. However, it requires a
write-path change and a spec addition for reserved snapshot-summary keys. It
also needs pointer compression or another size-control mechanism to avoid
repeating the full property map on every snapshot. The current PR avoids those
concerns by exposing property history already present in retained metadata
files.
* **Add a new top-level property-history field to `TableMetadata`.**:
Another option is to add a dedicated history structure to table metadata, such
as a list of property-map versions or property deltas. That would make the
history independent of metadata-log retention and could support direct point
lookups without reading historical metadata files. However, it would introduce
new persisted table metadata that every reader must preserve correctly, and it
would require defining retention, compatibility, and upgrade behavior for that
new structure. It is also heavier than necessary for this PR because Iceberg
already stores the complete property map in each retained `metadata.json`. A
read-only metadata table exposes that existing information without changing the
table metadata format.
## Testing
Run the following specific tests:
- `./gradlew :iceberg-spark:iceberg-spark-extensions-4.1_2.13:test --tests
"org.apache.iceberg.spark.extensions.TestMetadataTables.testTablePropertiesLog"`
- `./gradlew :iceberg-flink:iceberg-flink-2.1:test --tests
"org.apache.iceberg.flink.source.TestFlinkMetaDataTable.testTablePropertiesLog"`
## AI Assistance
I used Claude Code and Codex while preparing this PR. For the code changes,
I used them to explore the existing codebase and to review my implementation,
including checking whether it followed existing coding patterns, whether the
implementation and wording were consistent with nearby code, and whether there
were simpler alternatives. For the PR description, I used them as reviewers to
refine the text and point out missing context. I reviewed the final code and PR
text myself and am responsible for the submitted changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
