This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 06d0653f68 [docs] Explain Data Evolution design and file contracts
with diagrams (#9711)
06d0653f68 is described below
commit 06d0653f681ea4c832fe0ecbab9501c9d560dbdd
Author: Jingsong Lee <[email protected]>
AuthorDate: Thu Sep 10 13:07:57 2026 +0800
[docs] Explain Data Evolution design and file contracts with diagrams
(#9711)
---
docs/docs/multimodal-table/data-evolution.mdx | 806 +++++++++++++++-------
docs/static/img/data-evolution-blob-fallback.svg | 70 ++
docs/static/img/data-evolution-column-merge.svg | 78 +++
docs/static/img/data-evolution-file-layout.svg | 79 +++
docs/static/img/data-evolution-maintenance.svg | 92 +++
docs/static/img/data-evolution-partial-update.svg | 90 +++
docs/static/img/data-evolution-range-contract.svg | 82 +++
7 files changed, 1054 insertions(+), 243 deletions(-)
diff --git a/docs/docs/multimodal-table/data-evolution.mdx
b/docs/docs/multimodal-table/data-evolution.mdx
index e00f0785a3..02b1d27187 100644
--- a/docs/docs/multimodal-table/data-evolution.mdx
+++ b/docs/docs/multimodal-table/data-evolution.mdx
@@ -29,23 +29,34 @@ permissions and limitations under the License.
## Overview
-Paimon supports schema evolution, allowing you to add, modify, or delete column
-schema. Data Evolution mode extends this for append tables by allowing partial
-column updates without rewriting entire data files. Updated column data is
-written to new files and merged with the original data during reads.
-
-Data Evolution mode offers the following advantages:
-
-- **Efficient partial-column updates**: update a subset of columns and avoid
the
- I/O cost of rewriting untouched columns.
-- **Reduced file rewrites**: append new column data to dedicated files when
- backfilling or evolving data.
-- **Delete support**: record row deletions with deletion vectors without
- rewriting existing column files.
-- **Optimized reads**: merge original and updated column files at read time.
-
-To enable Data Evolution, create an append table with both
-`row-tracking.enabled` and `data-evolution.enabled` set to `true`.
+Data Evolution lets an append table update selected columns while keeping the
+files for untouched columns. A logical row can therefore be assembled from
+several physical files: its scalar fields from normal data files, its payload
+from BLOB files, and its embedding from dedicated vector files. Paimon aligns
+these pieces by **row ID**, and uses file versions to select the current
values.
+
+This is useful when a pipeline ingests data once and enriches it repeatedly,
+for example by adding captions, labels, or embeddings to images. Schema
+evolution changes which columns exist; Data Evolution writes or replaces their
+values for existing rows. Adding a column alone does not backfill it.
+
+The unit of a normal partial-column write is **the selected columns over a
+complete file row-ID range**. Updating one row can still rewrite those columns
+for every row in its file group. The saving comes from leaving the other
+columns, especially large payloads, untouched. Repeated updates add files and
+read work until compaction consolidates them.
+
+## Create a Table
+
+Set both `row-tracking.enabled` and `data-evolution.enabled` when creating the
+table. These options are immutable. Data Evolution requires an append table
+with no primary key and `bucket = -1` (the default for an append table).
+`clustering.incremental` cannot be enabled with Data Evolution.
+
+The following examples assume a configured Paimon catalog and the `default`
+database. Deletion vectors are enabled so the same table can also run the
+[delete examples](#deletes); they are optional if row-level deletes are not
+needed.
<Tabs groupId="data-evolution-create-table">
@@ -54,10 +65,11 @@ To enable Data Evolution, create an append table with both
```sql
CREATE TABLE target_table (id INT, b INT, c INT) TBLPROPERTIES (
'row-tracking.enabled' = 'true',
- 'data-evolution.enabled' = 'true'
+ 'data-evolution.enabled' = 'true',
+ 'deletion-vectors.enabled' = 'true'
);
-INSERT INTO target_table VALUES (1, 1, 1), (2, 2, 2);
+INSERT INTO target_table VALUES (1, 1, 10), (2, 2, 20);
```
</TabItem>
@@ -65,27 +77,29 @@ INSERT INTO target_table VALUES (1, 1, 1), (2, 2, 2);
<TabItem value="flink-sql" label="Flink SQL">
```sql
+SET 'execution.runtime-mode' = 'batch';
+
CREATE TABLE target_table (id INT, b INT, c INT) WITH (
'row-tracking.enabled' = 'true',
- 'data-evolution.enabled' = 'true'
+ 'data-evolution.enabled' = 'true',
+ 'deletion-vectors.enabled' = 'true'
);
-INSERT INTO target_table VALUES (1, 1, 1), (2, 2, 2);
+INSERT INTO target_table VALUES (1, 1, 10), (2, 2, 20);
```
+Wait for the initial insert job to finish before running an update.
+
</TabItem>
<TabItem value="python-api" label="Python API">
```python
import pyarrow as pa
-
from pypaimon import CatalogFactory, Schema
-catalog_options = {"warehouse": "/path/to/warehouse"}
-catalog = CatalogFactory.create(catalog_options)
+catalog = CatalogFactory.create({"warehouse": "/path/to/warehouse"})
catalog.create_database("default", True)
-
pa_schema = pa.schema([
("id", pa.int32()),
("b", pa.int32()),
@@ -96,338 +110,644 @@ schema = Schema.from_pyarrow_schema(
options={
"row-tracking.enabled": "true",
"data-evolution.enabled": "true",
+ "deletion-vectors.enabled": "true",
},
)
catalog.create_table("default.target_table", schema, False)
-
table = catalog.get_table("default.target_table")
+
builder = table.new_batch_write_builder()
write = builder.new_write()
commit = builder.new_commit()
-write.write_arrow(pa.Table.from_pydict(
- {"id": [1, 2], "b": [1, 2], "c": [1, 2]},
- schema=pa_schema,
-))
-commit.commit(write.prepare_commit())
-write.close()
-commit.close()
+try:
+ write.write_arrow(pa.Table.from_pydict(
+ {"id": [1, 2], "b": [1, 2], "c": [10, 20]}, schema=pa_schema,
+ ))
+ commit.commit(write.prepare_commit())
+finally:
+ write.close()
+ commit.close()
```
</TabItem>
</Tabs>
-### Compact Write-Column Metadata
-
-Tables with dedicated BLOB or vector files normally repeat the complete list of
-non-dedicated columns in every normal file's metadata. For wide tables, you can
-omit this redundant list by setting
-`data-evolution.write-cols-optimization.enabled` to `true`. A missing
-`writeCols` value then means that the file contains every non-dedicated column;
-dedicated files continue to record their columns explicitly.
-
-The option affects only new writes and is disabled by default. Readers support
-both representations without a read option. Before enabling it, upgrade every
-reader and maintenance job that accesses the table to a version that supports
-the compact representation, because older readers interpret a missing
-`writeCols` value as all table columns. Persist the option with `ALTER TABLE`
(or
-set it when creating the table) so that file schema versions record the compact
-representation. Before rolling readers back to an older version, rewrite files
-produced with this option or keep those readers upgraded.
+A business column such as `id` is not a primary key. Repeated `INSERT`
statements
+append new rows, even when business-key values repeat. `UPDATE` and `MERGE
INTO`
+use their predicates to find rows and use Paimon's `_ROW_ID` internally to
+write the changes.
-## Partial Updates
-
-You can update selected columns with Spark SQL `UPDATE` or `MERGE INTO`, the
-Flink `data_evolution_merge_into` procedure, or the PyPaimon table update API.
-Only the updated column files are written; untouched columns remain in their
-original files.
-
-<Tabs groupId="data-evolution-partial-update">
+For multimodal columns, see [BLOB](./blob) and [Vector](./vector) for their
type
+declarations and storage options. Dedicated BLOB/vector storage needs normal
+columns alongside it. A partial-column file containing ordinary columns is
+still a **normal file**; “dedicated” refers to the storage format, not to
+whether the file was created by an update.
-<TabItem value="spark-sql" label="Spark SQL">
+## File Group Spec
-```sql
-UPDATE target_table SET b = b + 10 WHERE id = 1;
+### Row IDs and File Metadata
-CREATE TABLE source_table (id INT, b INT);
-INSERT INTO source_table VALUES (1, 11), (2, 22);
+Within a Data Evolution file, logical row positions are contiguous and ordered.
+For a file with `firstRowId = s` and `rowCount = n`, its inclusive **RowId
Range**
+is:
-MERGE INTO target_table AS t
-USING source_table AS s
-ON t.id = s.id
-WHEN MATCHED THEN UPDATE SET t.b = s.b;
-
-SELECT * FROM target_table;
-+----+----+----+
-| id | b | c |
-+----+----+----+
-| 1 | 11 | 1 |
-| 2 | 22 | 2 |
-+----+----+----+
+```text
+R(file) = [s, s + n - 1]
+row ID at zero-based position i = s + i
```
-</TabItem>
+The row count includes positions hidden by deletion vectors. A logical delete
+does not shrink this range or shift the remaining positions.
-<TabItem value="flink-sql" label="Flink SQL">
+Paimon assigns new row IDs at commit time. A new normal file establishes the
+rows; dedicated files written alongside it receive the corresponding IDs,
+not additional row IDs. Partial updates reuse the existing IDs. Row IDs are
+independent of business keys, physical file names, and SQL result ordering.
-Flink does not currently support `MERGE INTO` syntax. Use the
-`data_evolution_merge_into` procedure instead:
+The metadata needed to reconstruct a row includes:
-```sql
-CREATE TABLE source_table (id INT, b INT);
-INSERT INTO source_table VALUES (1, 11), (2, 22);
+| Metadata | Contract |
+| --- | --- |
+| `firstRowId`, `rowCount` | Identify the complete logical row interval
represented by the file. |
+| `schemaId` | Identifies the schema used to write the file. Resolve its
columns against this schema, not just the latest table schema. |
+| `writeCols` | Lists the columns physically written, in their write order. An
omitted column supplies no new value. A missing list has a schema-dependent
meaning described [below](#compact-write-column-metadata). |
+| `minSequenceNumber`, `maxSequenceNumber` | Track file versions. New writes
receive the committing snapshot's sequence number; ordinary compaction
preserves the source version bounds. Normal files are considered in descending
`maxSequenceNumber` order. |
-CALL sys.data_evolution_merge_into(
- 'default.target_table',
- '',
- '',
- 'source_table',
- 'source_table.id=target_table.id',
- 'b=source_table.b',
- 2
-);
+Column names in `writeCols` are resolved through the file's schema to field
+IDs. This allows readers to distinguish a column that was renamed from a new
+column added after the file was written.
-SELECT * FROM target_table;
-+----+----+----+
-| id | b | c |
-+----+----+----+
-| 1 | 11 | 1 |
-| 2 | 22 | 2 |
-+----+----+----+
-```
+### Normal Files: Equal or Disjoint Ranges
-</TabItem>
+Normal files use the table's ordinary `file.format`, such as Parquet or ORC.
+They can contain all normal columns or only the columns selected by an update.
+For two live normal files, their RowId Ranges must be either **identical** or
+**disjoint**. Partial overlap and strict containment are both invalid.
-<TabItem value="python-api" label="Python API">
+Normal files with the same range form the normal part of a file group. They
+must have both the same `firstRowId` and the same `rowCount`. Matching only
+`firstRowId` is insufficient.
-PyPaimon exposes an UPDATE-like table update API. It accepts a `Predicate`
-for the `WHERE` condition and literal assignments for the `SET` clause.
+For example, after updating `b` in a normal file covering `[0, 5]`, the new
+normal file for `b` must also cover `[0, 5]`. It cannot contain only `[2, 3]`,
+even if the update predicate matched only those two rows. The writer supplies
+the old values of `b` for the other positions.
-```python
-table = catalog.get_table("default.target_table")
+### Dedicated Files: Contained Within One Normal Range
-builder = table.new_batch_write_builder()
-table_update = builder.new_update()
-predicate = table_update.new_predicate_builder().is_in("id", [1, 2])
+Dedicated files store columns in specialized formats: a BLOB file belongs to
+one BLOB field, while a dedicated vector file can contain a set of vector
+fields. Their file boundaries can differ from normal-file boundaries.
-messages = table_update.update_by_predicate(predicate, {"b": 100})
+For **every live dedicated file**, there must be a normal file in the same
+partition whose range contains the dedicated file's **entire** range:
-commit = builder.new_commit()
-commit.commit(messages)
-commit.close()
+```text
+R(dedicated) is a subset of R(normal)
+normal.firstRowId <= dedicated.firstRowId
+dedicated.lastRowId <= normal.lastRowId
```
-</TabItem>
+Equality is allowed. The reverse containment is not sufficient. A dedicated
+file must not span two adjacent normal ranges, even if their union covers all
+its rows. Consequently, one normal range can own many smaller dedicated
+files, but each dedicated file belongs to exactly one distinct normal range.
+
+During an append, dedicated writers may roll into smaller files independently.
+When the normal writer closes its file, the associated dedicated writers close
+as well. For each written BLOB field, the sum of its dedicated file row counts
+matches the normal file's row count; the same holds for the written vector
+file set. This maintains row alignment without forcing large payloads into
+one physical file.
+
+For a complete vector-column read, the selected vector files must concatenate
+to the normal group's full range in row-ID order, without gaps. BLOB reads
+also support partial coverage: gaps in a newer BLOB version mean “look in an
+older version,” and a row with no value in any version reads as `NULL`. This
+supports backfilling a newly added nullable BLOB column for only some rows.
+
+### Valid and Invalid Layouts
+
+The following is one valid snapshot. Both normal files in group A cover all
+six positions; its BLOB files cover smaller intervals. Group B is adjacent
+and independent.
+
+
+
+The normal/dedicated distinction changes which ranges are legal. Each
+candidate in the next figure is checked separately against the same two
+normal ranges; the candidates are not files committed together.
+
+
+
+With normal ranges `[0, 5]` and `[6, 9]`:
+
+| Proposed file | Valid? | Reason |
+| --- | --- | --- |
+| Normal update `[0, 5]` | Yes | Exactly matches group A. |
+| Normal update `[0, 3]` | No | Shares the start, but not the full range. |
+| Normal update `[2, 5]` | No | A normal update cannot cover only a subset. |
+| Normal update `[4, 8]` | No | Partially overlaps existing normal ranges. |
+| Dedicated file `[2, 4]` | Yes | Entirely contained in group A; its column's
coverage rules must also hold. |
+| Dedicated file `[0, 5]` | Yes | Equality with a normal range is allowed. |
+| Dedicated file `[4, 7]` | No | Crosses from group A to group B. |
+| Dedicated file `[0, 9]` | No | The union of two normal ranges is not one
owning range. |
+| Dedicated file `[10, 12]` | No | No normal file establishes those rows. |
+
+These constraints apply to the **live files in a snapshot**, not to all files
+still retained on storage. Compaction can replace both groups with a new
+normal range `[0, 9]` atomically. Once the old normal files have been removed
+from the new snapshot, existing dedicated files inside `[0, 9]` remain valid,
+and a dedicated file covering `[0, 9]` can be valid too.
+
+### How Reads Reconstruct Rows
+
+A reader first groups live files by overlapping RowId Ranges. Under the above
+contracts, each group has one distinct normal range and its contained
+dedicated files. It then:
+
+1. Resolves each file's written columns using its schema version.
+2. Selects the newest normal file that supplies each requested column. An
+ older file is still needed if it supplies another requested column.
+3. Concatenates the selected vector files and resolves BLOB versions for the
+ same logical row positions.
+4. Combines the selected columns by row position and applies logical deletions
+ consistently across the group. Columns absent from all files are filled
+ with `NULL` if nullable; a missing non-nullable column is an error.
+
+This is a positional column merge, not a join on a user key. Readers can skip
+column files that contribute no requested values. Filtering and index pruning
+must use the current column providers: statistics in an old file cannot be
+used to discard rows based on a column that a newer file has overwritten.
+
+For example, these normal files all cover `[0, 2]`:
+
+
+
+The result is `(10, 1, 101), (20, 22, 200), (30, 3, 300)`. N1 replaces `b`
+for the whole range, including the preserved values for rows 0 and 2. N2
+does not replace `b`, because `b` is absent from its written columns.
+
+There are three different meanings to keep separate:
+
+| Representation | Meaning during a read |
+| --- | --- |
+| Column absent from a normal file's written schema | This file does not
update the column; use another provider. |
+| Explicit `NULL` in a written column | The current value is `NULL`; do not
fall back to an older non-null value. |
+| Internal BLOB placeholder | Preserve the older BLOB value for this row.
Continue through older BLOB versions until a non-placeholder value, including
explicit `NULL`, is found. |
+
+BLOB placeholders let an update preserve unchanged payloads without copying
+them into the new BLOB file. They are managed by the writer; applications
+should not substitute `NULL` for “leave unchanged.” Descriptor-only and BLOB
+view fields stored inline follow normal-column storage rules.
+
+
-</Tabs>
+## Partial Updates
-Notes:
-
-- Spark SQL supports standalone `UPDATE` statements for Data Evolution tables.
-- Concurrent Spark SQL `UPDATE` statements that update the same data file and
- columns may be retried automatically. Configure retry attempts with
- `spark.paimon.write.data-evolution.update-conflict-retry.max-attempts` and
- retry wait time with
- `spark.paimon.write.data-evolution.update-conflict-retry.wait-ms`.
-- If concurrent compaction changes row-ID file boundaries after Spark SQL
- `MERGE INTO` stages regular partial-column files, Spark rebases those staged
- files onto the latest boundaries before committing instead of rerunning the
- MERGE source and join. Spark performs this rewrite with distributed DataFrame
- processing, so the PyPaimon-only
- `data-evolution.row-id-conflict-rewrite.max-size` limit does not apply.
- Recovery does not apply when deletion vectors are enabled or when
- existing-row BLOB or VECTOR files are staged, and it does not hide logical
- concurrent-update conflicts.
-- In Spark SQL, `MERGE INTO` supports `WHEN NOT MATCHED BY SOURCE` for delete
- actions on Data Evolution tables.
-- The Flink `data_evolution_merge_into` procedure currently supports updating
- or inserting columns, but not inserting new rows.
+For a normal-column update, the write path preserves the file contract through
+the following steps:
-## Deletes
+1. Pin a target snapshot, find the matching row IDs, and map them to normal
+ file ranges in that snapshot.
+2. For each affected range, produce the selected columns in row-ID order:
+ new values for matched rows and existing values for unmatched rows. Keep
+ deleted positions aligned as well.
+3. Write a normal partial file with the original range's `firstRowId` and
+ `rowCount`. Normal-file rolling is disabled for this write, because a
+ smaller output would violate the equal-range rule.
+4. Commit the new files atomically with conflict checks. Readers of the new
+ snapshot see the updated column versions; older snapshots keep their
+ previous view.
-Data Evolution tables can use deletion vectors to record deleted rows without
-rewriting existing column files. To write deletes, enable deletion vectors
-together with row tracking and Data Evolution:
+The engine and update APIs perform this alignment. Applications specify the
+matching rows and values; they do not need to construct file groups themselves.
-:::warning
+
-Deleting rows with deletion vectors does not physically remove the original
-data immediately. Data Evolution compaction preserves row IDs and logical
-deletions, so it does not materialize deleted rows. The legacy
-`data-evolution.compaction.rewrite-row-ids` option is no longer supported. Use
-the `materialize_deletion_vectors` procedure to apply deletion vectors to the
-latest table state and replace the affected files. Historical snapshots and
-tags may still reference the replaced files; reclaiming their storage requires
-those references to expire and snapshot expiration to remove the files.
+Here `id = 20` selects only row ID 1, but N1 still contains three values of
+`b`. The row-ID labels show alignment; they are not extra business columns.
-:::
+### Spark SQL
-Both Flink and Spark expose the same procedure:
+Spark supports `UPDATE` expressions and `MERGE INTO`. For example:
```sql
-CALL sys.materialize_deletion_vectors(`table` => 'default.target_table');
-```
+UPDATE target_table SET b = b + 10 WHERE id = 1;
-The procedure can limit the rewrite to selected partitions. `partitions` and
-`where` cannot be used together:
+CREATE TABLE source_table (id INT, b INT, c INT);
+INSERT INTO source_table VALUES (1, 100, 1000), (3, 300, 3000);
-```sql
-CALL sys.materialize_deletion_vectors(
- `table` => 'default.target_table',
- partitions => 'dt=2026-08-12');
+MERGE INTO target_table AS t
+USING source_table AS s
+ON t.id = s.id
+WHEN MATCHED THEN UPDATE SET t.b = s.b
+WHEN NOT MATCHED THEN INSERT (id, b, c) VALUES (s.id, s.b, s.c);
-CALL sys.materialize_deletion_vectors(
- `table` => 'default.target_table',
- `where` => 'dt >= 20260801');
+SELECT * FROM target_table ORDER BY id;
+-- (1, 100, 10), (2, 2, 20), (3, 300, 3000)
```
-Materialization rewrites the affected data, assigns new row IDs to surviving
-rows, removes the applied deletion vectors, and drops affected global indexes.
-Consumers which persist `_ROW_ID` values must therefore coordinate with this
-operation. Concurrent changes to affected row-ID ranges cause the procedure to
-fail instead of committing stale results. Materialization of vector-store files
-is not currently supported.
+The matched update writes `b` over the affected existing normal ranges and
+keeps their row IDs. The unmatched insert writes a new row and receives a new
+row ID. Assign only the columns you intend to change; assigning every column
+can remove the I/O benefit of a partial update. Partition columns cannot be
+changed by a matched partial update.
-Spark processes bounded batches until all matching deletion vectors have been
-materialized. To keep a Flink job bounded, one Flink procedure invocation
-processes one batch with a soft target of 100,000 deletion vectors. An
-overlapping row-ID component is never split and can exceed the target. Invoke
-the Flink procedure repeatedly until an invocation makes no changes.
+### Flink SQL
+
+Use the `data_evolution_merge_into` procedure for matched updates. It runs in
+batch mode and can populate an existing column, including a newly added
+column, but does not append unmatched source rows or perform deletes. The
+target must already contain data.
```sql
-CREATE TABLE target_table (id INT, b INT, c INT) TBLPROPERTIES (
- 'row-tracking.enabled' = 'true',
- 'data-evolution.enabled' = 'true',
- 'deletion-vectors.enabled' = 'true'
+CREATE TABLE source_table (id INT, b INT);
+INSERT INTO source_table VALUES (1, 100), (2, 200);
+
+-- Wait for the source insert to finish before calling the procedure.
+CALL sys.data_evolution_merge_into(
+ 'default.target_table', -- target table
+ 't', -- target alias
+ '', -- optional source SQL statements
+ 'source_table', -- source table or view
+ 't.id=source_table.id', -- match condition
+ 'b=source_table.b', -- columns and expressions to write
+ 2 -- sink parallelism
);
```
-Spark SQL supports `DELETE FROM` for Data Evolution tables:
+Use `''` for unused optional positional arguments. Updating partition columns
+is not supported. For BLOB updates, this Flink procedure supports
+[descriptor-only](./blob#descriptor-only-storage) and [BLOB
view](./blob#blob-view)
+fields; it rejects raw-data BLOB fields, including BLOB arrays and maps.
-```sql
-DELETE FROM target_table WHERE id = 1;
-```
+### Python API
-Spark SQL also supports delete actions in `MERGE INTO`:
+PyPaimon returns commit messages from its update APIs. Commit them to make the
+change visible:
-```sql
-CREATE TABLE source_table (id INT, op STRING);
-INSERT INTO source_table VALUES (2, 'delete'), (3, 'update');
+```python
+table = catalog.get_table("default.target_table")
+builder = table.new_batch_write_builder()
+update = builder.new_update()
+predicate = update.new_predicate_builder().is_in("id", [1, 2])
+messages = update.update_by_predicate(predicate, {"b": 100})
-MERGE INTO target_table AS t
-USING source_table AS s
-ON t.id = s.id
-WHEN MATCHED AND s.op = 'delete' THEN DELETE
-WHEN MATCHED AND s.op = 'update' THEN UPDATE SET t.b = t.b + 10
-WHEN NOT MATCHED BY SOURCE AND t.id > 10 THEN DELETE;
+commit = builder.new_commit()
+try:
+ commit.commit(messages)
+finally:
+ commit.close()
```
-The `WHEN NOT MATCHED BY SOURCE` clause requires Spark 3.4 or later.
+For expressions, row-ID updates, business-key upserts, `merge_into`, and stream
+writer lifecycles, see [PyPaimon Data Evolution](../pypaimon/data-evolution).
+These APIs handle file alignment; writing an arbitrary small normal file and
+assigning it an existing row ID does not satisfy the file contract.
+
+## Self Updates
+
+A self update derives new values from the table's current rows. It is also the
+usual backfill workflow: first add a nullable column, then compute its values.
+The examples in this section are alternatives to run after creating the table.
+
+<Tabs groupId="data-evolution-self-update">
-With Flink 1.17 or later, SQL supports `DELETE FROM` for Data Evolution tables
-in batch mode. For row-level predicates, matching rows are recorded in deletion
-vectors, so data files are not rewritten:
+<TabItem value="spark-sql" label="Spark SQL">
```sql
-DELETE FROM target_table WHERE id = 1;
-```
+ALTER TABLE target_table ADD COLUMN total INT;
-Flink users can also submit the
-[`delete` action](../flink/action-jars#deleting-from-a-data-evolution-table).
+UPDATE target_table SET total = b + c;
-## Self Updates
+SELECT id, total FROM target_table ORDER BY id;
+```
-Self updates transform existing column values in place. In Flink SQL, create a
-temporary source view from the `$row_tracking` system table and join by
-`_ROW_ID`. In PyPaimon, use the shard scan + rewrite workflow to read existing
-values and write the derived columns back.
+This writes `total` for the existing ranges. It leaves `id`, `b`, and `c` in
+their existing column files.
-<Tabs groupId="data-evolution-self-update">
+</TabItem>
<TabItem value="flink-sql" label="Flink SQL">
```sql
+ALTER TABLE target_table ADD total INT;
+
CREATE TEMPORARY VIEW source_view AS
-SELECT _ROW_ID, b + c AS b
-FROM default.target_table$row_tracking;
+SELECT _ROW_ID, b + c AS total
+FROM default.`target_table$row_tracking`;
CALL sys.data_evolution_merge_into(
'default.target_table',
- 'TempT',
+ 't',
'',
'source_view',
- 'TempT._ROW_ID=source_view._ROW_ID',
- 'b=source_view.b',
+ 't._ROW_ID=source_view._ROW_ID',
+ 'total=source_view.total',
2
);
```
+The temporary view supplies the source because the procedure does not allow
+identical source and target table names. The `_ROW_ID` equality identifies
+the self-merge pattern. It supports matched updates only.
+
</TabItem>
<TabItem value="python-api" label="Python API">
-```python
-import pyarrow as pa
+For a derived-column workflow, read a shard and write the computed values in
+exactly the same order. This example replaces the existing `b` column:
+```python
table = catalog.get_table("default.target_table")
-
builder = table.new_batch_write_builder()
-table_update = builder.new_update()
-table_update.with_read_projection(["b", "c"])
-table_update.with_update_type(["b"])
+update = builder.new_update()
+update.with_read_projection(["b", "c"])
+update.with_update_type(["b"])
-updater = table_update.new_shard_updator(0, 1)
+updater = update.new_shard_updator(0, 1)
reader = updater.arrow_reader()
for batch in iter(reader.read_next_batch, None):
b = batch.column("b").to_pylist()
c = batch.column("c").to_pylist()
updater.update_by_arrow_batch(pa.RecordBatch.from_pydict(
- {"b": [bi + ci for bi, ci in zip(b, c)]},
+ {"b": [None if bi is None or ci is None else bi + ci
+ for bi, ci in zip(b, c)]},
schema=pa.schema([("b", pa.int32())]),
))
messages = updater.prepare_commit()
commit = builder.new_commit()
-commit.commit(messages)
-commit.close()
+try:
+ commit.commit(messages)
+finally:
+ commit.close()
```
+Each output batch must have the same row count and order as its input batch.
+Do not filter, sort, or duplicate rows between the read and update steps.
+Multiple workers can use distinct shard indices with the same shard count.
+
</TabItem>
</Tabs>
-Self-update notes:
+For explicit row-ID inspection in SQL, query the system table:
+
+```sql
+SELECT _ROW_ID, id, b, c FROM default.`target_table$row_tracking`;
+```
+
+Obtain row IDs from the table instead of deriving them from a business key or
+from result order. Ordinary updates and compaction preserve them, but
+[maintenance that reassigns row IDs](#row-id-lifetime) does not.
-- The source and target table name cannot be the same in the Flink procedure.
- Create a temporary view as the source.
-- Use `view._ROW_ID = source._ROW_ID` to identify the self-merge pattern in
- Flink.
-- `_ROW_ID` is only available via the `$row_tracking` system table in SQL.
-- Self-merge only supports `WHEN MATCHED THEN UPDATE` semantics.
+### Nested Fields
-## File Group Spec
+By default, the physical update unit is a top-level column, including a whole
+`ROW`/`STRUCT` column. To write only direct struct subfields, persist
+`data-evolution.nested-field.enabled = true` on the table. Written-column
+metadata can then contain paths such as `payload.caption`, and readers select
+the latest provider for the corresponding subfields. Row alignment still
+covers the full normal-file range.
+
+The current implementation supports composing one level of struct subfields
+across files. It does not support splitting a deeper sub-struct across
+multiple files. Engine support for assignment syntax also matters; for
+example, the Flink procedure requires the option for a subfield assignment.
+
+:::warning
+
+Upgrade every reader, writer, compactor, and maintenance job before enabling
+nested-field evolution. Enable it through a persisted table-option change;
+dynamic overrides and later disabling or removing the option are unsupported.
+Older components cannot reconstruct files containing nested write paths, so a
+binary downgrade is unsafe once these files have been committed.
+
+:::
+
+## Deletes
+
+Row-level deletes require `deletion-vectors.enabled = true`. A deletion vector
+records deleted positions against a normal anchor file in the group (the
+oldest normal file, ordered by version and then file name). Readers apply
+these deletions to all column providers for the same logical rows. Deleting
+a row therefore hides its scalar, BLOB, and vector values together without
+rewriting those data files.
+
+Spark SQL supports:
+
+```sql
+DELETE FROM target_table WHERE id = 1;
+```
+
+Spark can also combine update and delete clauses in `MERGE INTO`:
+
+```sql
+CREATE TABLE delete_source (id INT, op STRING);
+INSERT INTO delete_source VALUES (2, 'delete'), (3, 'update');
+
+MERGE INTO target_table AS t
+USING delete_source AS s
+ON t.id = s.id
+WHEN MATCHED AND s.op = 'delete' THEN DELETE
+WHEN MATCHED AND s.op = 'update' THEN UPDATE SET t.b = t.b + 10
+WHEN NOT MATCHED BY SOURCE AND t.id > 10 THEN DELETE;
+```
+
+`WHEN NOT MATCHED BY SOURCE` requires Spark 3.4 or later.
+
+Flink 1.17 or later supports `DELETE FROM` for Data Evolution tables in batch
+mode. Flink also provides the
+[`delete` action](../flink/action-jars#deleting-from-a-data-evolution-table).
+PyPaimon provides `delete_by_predicate` and `delete_by_row_id`; see its
+[delete examples](../pypaimon/data-evolution#delete-rows). Partition-only
deletes
+can use a partition-removal path instead of row-level deletion vectors.
+
+Deleting a row leaves its position in the file range occupied but invisible.
+For example, deleting row ID 2 from `[0, 5]` leaves the metadata range `[0,
5]`,
+with five visible rows. An update must not close that gap or shift rows 3–5.
+
+## Compaction and Row-ID Lifetime
+
+
+
+The new IDs `[100, 102]` in this example are illustrative. Actual IDs are
+allocated at commit time. Both paths preserve the visible values, but only
+materialization removes deleted positions from the rewritten files.
+
+### Ordinary Compaction
+
+Repeated partial updates leave multiple column versions in each group.
+Ordinary Data Evolution compaction reads their merged values and writes
+consolidated normal files. It can also merge adjacent ranges in the same
+partition. It preserves row IDs, row order, version information, and logical
+deletions, and keeps the resulting normal/dedicated layout within the file
+contract.
+
+For example, replacing all normal files for `[0, 5]` and `[6, 9]` with one
+normal file `[0, 9]` changes file boundaries without changing any row's ID.
+Dedicated files contained in either old range are contained in the new range,
+so they do not need rewriting just because the normal files were merged.
+
+Both Spark and Flink provide:
+
+```sql
+CALL sys.compact(`table` => 'default.target_table');
+```
+
+BLOB compaction is separately controlled by `blob-compaction.enabled`
+(default `false`). Dedicated vector-file compaction is not currently
+supported. See [Dedicated Compaction](../maintenance/dedicated-compaction)
+for scheduling compaction jobs.
-Through the row-id metadata, files are organized into file groups.
+Ordinary compaction **does not physically remove rows hidden by deletion
+vectors**. Removing their positions would change the alignment with untouched
+column files.
-When writing, the Data Evolution update path writes only the specified updated
-columns to new files. The original data files remain unchanged.
+### Materialize Deletion Vectors
-When reading, Paimon reads both the original data files and the new files
-containing updated column data, then merges files with the same `first row id`
-to present a unified view of the table.
+To remove deleted positions from the latest table state, run:
-After writing, files in `target_table` are organized as below:
+```sql
+CALL sys.materialize_deletion_vectors(`table` => 'default.target_table');
+```
-
+This operation reads the affected groups with their deletions applied,
+rewrites the surviving data, assigns new row IDs, removes the applied deletion
+vectors, and drops affected global indexes. Concurrent changes to the affected
+row-ID ranges cause the operation to fail instead of committing stale results.
+Materialization of groups containing dedicated vector files is not currently
+supported.
-When reading, files with the same `first row id` are merged:
+For a partitioned table, select partitions by either `partitions` or a
+partition predicate in `where`; they cannot be used together:
+
+```sql
+CALL sys.materialize_deletion_vectors(
+ `table` => 'default.partitioned_table',
+ partitions => 'dt=2026-08-12');
+
+CALL sys.materialize_deletion_vectors(
+ `table` => 'default.partitioned_table',
+ `where` => 'dt >= 20260801');
+```
-
+Spark processes bounded batches until all matching deletion vectors are
+materialized. Each Flink invocation processes one batch with a soft target of
+100,000 deletion vectors. An overlapping row-ID component is never split and
+can exceed that target. Repeat the Flink call until it makes no changes.
+
+Historical snapshots and tags can still reference the replaced files.
+Reclaiming their storage requires those references to expire and snapshot
+expiration to remove the files. The legacy
+`data-evolution.compaction.rewrite-row-ids = true` setting is rejected; use the
+materialization procedure instead.
+
+### Row-ID Lifetime
+
+| Operation | Effect on row IDs |
+| --- | --- |
+| Append new rows | Allocates new IDs. |
+| Partial update or column backfill | Preserves existing IDs. |
+| Logical delete | Hides IDs without shifting the remaining rows. |
+| Ordinary Data Evolution compaction | Preserves IDs, but may change
file-group boundaries. |
+| Deletion-vector materialization | Assigns new IDs to surviving rows in the
rewritten ranges. |
+| Explicit `reassign_row_id` maintenance | Changes IDs by remapping file
metadata. |
+| Overwrite or partition removal | Removes the replaced rows from the current
table state; do not assume their IDs remain usable. |
+
+Applications that persist `_ROW_ID` values, including references from other
+tables, must coordinate with operations that replace or reassign IDs. Row IDs
+are not permanent application identifiers. The `reassign_row_id` procedure is
+documented in the [Spark](../spark/procedures) and
+[Flink](../flink/procedures) procedure references.
+
+## Concurrent Writes
+
+Updates are planned against a snapshot and checked when committing. Paimon
+checks both the layout contract and whether intervening writes conflict with
+the staged changes.
+
+- A new normal partial file must exactly match a current normal range; an
+ existing-row dedicated file must be contained within one current normal
+ range. A removed row range cannot be updated using stale metadata.
+- Logical update conflicts consider both overlapping row ranges and written
+ field IDs. Updating different columns can avoid this conflict. Updating the
+ same column in the same normal range can conflict even when the predicates
+ selected different rows, because each staged file covers the full range.
+- A concurrent compaction can change boundaries while preserving the row IDs.
+ This is a layout conflict and is distinct from another writer changing the
+ same column values.
+
+Spark `UPDATE` can retry update conflicts. Configure it with
+`spark.paimon.write.data-evolution.update-conflict-retry.max-attempts` and
+`spark.paimon.write.data-evolution.update-conflict-retry.wait-ms`.
+
+Spark `MERGE INTO` can rebase staged normal partial-column files onto the
+latest boundaries after concurrent compaction, using distributed processing
+without rerunning the source and join. PyPaimon has similar recovery bounded
+by `data-evolution.row-id-conflict-rewrite.max-size` (default `256 MB`, `0 B`
+disables it), measured over the affected current data files. This size limit
+does not apply to Spark.
+
+This boundary recovery is not attempted when deletion vectors are enabled or
+when the commit contains existing-row BLOB or VECTOR staged files. It does
+not suppress logical concurrent-update conflicts. If recovery is unavailable,
+re-plan the operation against the latest snapshot.
+
+## File Sizing and Metadata Compatibility
+
+`target-file-size` controls normal-file sizing. `blob.target-file-size` and
+`vector.target-file-size` control dedicated file sizing and default to the
+normal target size. These are size targets, not permission to break the
+containment rules.
+
+A large normal range makes even a selective normal-column update process many
+rows. `target-file-row-num` can bound newly appended file row counts where the
+writer supports it; it is disabled by default. It does not split existing
+normal ranges during partial updates, and Data Evolution compaction can
+produce larger ranges. Choose file sizing with both scan efficiency and
+future update cost in mind.
-The advantages of this mode are:
+### Compact Write-Column Metadata
+
+With `data-evolution.write-cols-optimization.enabled = true`, a normal file
+that contains **all non-dedicated columns** can omit the repeated `writeCols`
+list. Partial subsets and dedicated files keep explicit written columns.
+The option is disabled by default and affects new writes.
+
+A reader interprets a missing list using the file's `schemaId`:
+
+- For a schema without this optimization, missing `writeCols` means the full
+ schema used to write that file.
+- For a schema with the optimization and dedicated columns, it means all
+ non-dedicated columns in that schema.
+
+Persist this option with `CREATE TABLE` or `ALTER TABLE` so the file schema
+records the interpretation. Upgrade all readers and maintenance jobs before
+enabling it: older readers treat a missing list as all table columns. Readers
+support both encodings without a read option. Before rolling back to an older
+reader, rewrite the files produced with this option or keep the readers
+upgraded.
+
+### Inspect the Layout
+
+Use the `$files` system table to inspect the live file metadata:
+
+```sql
+SELECT file_path, schema_id, write_cols,
+ first_row_id,
+ first_row_id + record_count - 1 AS last_row_id,
+ record_count, max_sequence_number
+FROM default.`target_table$files`
+ORDER BY first_row_id, max_sequence_number;
+```
-- Avoid rewriting the whole file when updating partial columns, reducing I/O
- cost.
-- Keep read performance efficient through optimized merge processing.
-- Use disk space more efficiently because only updated columns are written to
- new files.
+Compare both endpoints when checking normal-file alignment, and compare each
+dedicated range against a **single** normal range. Summing `record_count`
+across all files overcounts logical rows because column versions and dedicated
+files can represent the same positions; deletion vectors reduce visible rows
+further. Use a table query for the logical row count.
diff --git a/docs/static/img/data-evolution-blob-fallback.svg
b/docs/static/img/data-evolution-blob-fallback.svg
new file mode 100644
index 0000000000..f55ab948ef
--- /dev/null
+++ b/docs/static/img/data-evolution-blob-fallback.svg
@@ -0,0 +1,70 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="413" viewBox="0 0
900 413" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">BLOB: a placeholder preserves; NULL replaces</title>
+<desc id="desc">For row 0, a newer placeholder falls back to old image A. For
row 1, newer explicit NULL overrides old image B. For row 2, new image Z
overrides old image C. The reader stops at the first non-placeholder value,
even when that value is NULL.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="898" height="411" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">BLOB: a placeholder preserves; NULL replaces</text>
+<text x="28" y="76" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">For each row, inspect BLOB versions from newest to oldest.</text>
+<rect x="28" y="103" width="88" height="43" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.0" y="131" font-size="18" font-weight="700" text-anchor="middle"
fill="#172b4d">Row ID</text>
+<rect x="116" y="103" width="225" height="43" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="228.5" y="131" font-size="18" font-weight="700" text-anchor="middle"
fill="#172b4d">Newer version</text>
+<rect x="341" y="103" width="199" height="43" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="440.5" y="131" font-size="18" font-weight="700" text-anchor="middle"
fill="#172b4d">Older version</text>
+<rect x="540" y="103" width="152" height="43" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="616.0" y="131" font-size="18" font-weight="700" text-anchor="middle"
fill="#172b4d">Result</text>
+<rect x="692" y="103" width="180" height="43" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="782.0" y="131" font-size="18" font-weight="700" text-anchor="middle"
fill="#172b4d">Why?</text>
+<rect x="28" y="146" width="88" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.0" y="181" font-size="19" font-weight="400" text-anchor="middle"
fill="#172b4d">0</text>
+<rect x="116" y="146" width="225" height="57" rx="0" fill="#f1eafa"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="228.5" y="181" font-size="19" font-weight="700" text-anchor="middle"
fill="#7140ad">PLACEHOLDER</text>
+<rect x="341" y="146" width="199" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="440.5" y="181" font-size="19" font-weight="400" text-anchor="middle"
fill="#172b4d">image A</text>
+<rect x="540" y="146" width="152" height="57" rx="0" fill="#f1eafa"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="616.0" y="181" font-size="19" font-weight="700" text-anchor="middle"
fill="#7140ad">image A</text>
+<rect x="692" y="146" width="180" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="782.0" y="181" font-size="18" font-weight="400" text-anchor="middle"
fill="#172b4d">fall back</text>
+<rect x="28" y="203" width="88" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.0" y="238" font-size="19" font-weight="400" text-anchor="middle"
fill="#172b4d">1</text>
+<rect x="116" y="203" width="225" height="57" rx="0" fill="#fff0f1"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="228.5" y="238" font-size="19" font-weight="700" text-anchor="middle"
fill="#b42332">NULL</text>
+<rect x="341" y="203" width="199" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="440.5" y="238" font-size="19" font-weight="400" text-anchor="middle"
fill="#526277">image B</text>
+<rect x="540" y="203" width="152" height="57" rx="0" fill="#fff0f1"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="616.0" y="238" font-size="19" font-weight="700" text-anchor="middle"
fill="#b42332">NULL</text>
+<rect x="692" y="203" width="180" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="782.0" y="238" font-size="18" font-weight="400" text-anchor="middle"
fill="#172b4d">stop at NULL</text>
+<rect x="28" y="260" width="88" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.0" y="295" font-size="19" font-weight="400" text-anchor="middle"
fill="#172b4d">2</text>
+<rect x="116" y="260" width="225" height="57" rx="0" fill="#f1eafa"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="228.5" y="295" font-size="19" font-weight="700" text-anchor="middle"
fill="#7140ad">image Z</text>
+<rect x="341" y="260" width="199" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="440.5" y="295" font-size="19" font-weight="400" text-anchor="middle"
fill="#526277">image C</text>
+<rect x="540" y="260" width="152" height="57" rx="0" fill="#f1eafa"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="616.0" y="295" font-size="19" font-weight="700" text-anchor="middle"
fill="#7140ad">image Z</text>
+<rect x="692" y="260" width="180" height="57" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="782.0" y="295" font-size="18" font-weight="400" text-anchor="middle"
fill="#172b4d">use new value</text>
+<rect x="28" y="343" width="844" height="45" rx="8" fill="#f1eafa"
stroke="#f1eafa" stroke-width="1.5"/>
+<text x="44" y="370" font-size="18" font-weight="400" text-anchor="start"
fill="#7140ad">Only a placeholder falls back. An explicit NULL is already the
new value.</text>
+</g>
+</svg>
diff --git a/docs/static/img/data-evolution-column-merge.svg
b/docs/static/img/data-evolution-column-merge.svg
new file mode 100644
index 0000000000..fc9e364dbb
--- /dev/null
+++ b/docs/static/img/data-evolution-column-merge.svg
@@ -0,0 +1,78 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="564" viewBox="0 0
900 564" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">Read each column from its newest provider</title>
+<desc id="desc">Three normal files all cover row IDs 0 to 2. N0 version 1
contains id, b, c; N1 version 2 contains b; N2 version 3 contains c. The result
takes id from N0, b from N1, and c from N2, combining them at matching row
positions.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="898" height="562" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">Read each column from its newest provider</text>
+<text x="28" y="76" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">All three files cover [0, 2]. Newer files override only the
columns they contain.</text>
+<rect x="28" y="101" width="294" height="139" rx="8" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="44" y="131" font-size="22" font-weight="700" text-anchor="start"
fill="#2463b4">N0 · id, b, c</text>
+<text x="44" y="158" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">version 1</text>
+<rect x="348" y="101" width="242" height="139" rx="8" fill="#fff2d7"
stroke="#9a5200" stroke-width="1.5"/>
+<text x="364" y="131" font-size="22" font-weight="700" text-anchor="start"
fill="#9a5200">N1 · b</text>
+<text x="364" y="158" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">version 2</text>
+<rect x="616" y="101" width="256" height="139" rx="8" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="632" y="131" font-size="22" font-weight="700" text-anchor="start"
fill="#10705d">N2 · c</text>
+<text x="632" y="158" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">version 3</text>
+<text x="44" y="187" font-size="20" font-weight="700" text-anchor="start"
fill="#2463b4">id: 10, 20, 30</text>
+<text x="44" y="211" font-size="17" font-weight="400" text-anchor="start"
fill="#526277">b, c superseded by N1, N2</text>
+<text x="364" y="198" font-size="21" font-weight="700" text-anchor="start"
fill="#9a5200">b: 1, 22, 3</text>
+<text x="632" y="198" font-size="21" font-weight="700" text-anchor="start"
fill="#10705d">c: 101, 200, 300</text>
+<path d="M 175 242 L 175 269 L 329 269 L 329 296" fill="none" stroke="#526277"
stroke-width="2" marker-end="url(#arrow)"/>
+<path d="M 469 242 L 469 269 L 495 269 L 495 296" fill="none" stroke="#526277"
stroke-width="2" marker-end="url(#arrow)"/>
+<path d="M 744 242 L 744 269 L 661 269 L 661 296" fill="none" stroke="#526277"
stroke-width="2" marker-end="url(#arrow)"/>
+<rect x="136" y="300" width="110" height="41" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="191.0" y="328" font-size="19" font-weight="700" text-anchor="middle"
fill="#526277">Row ID</text>
+<rect x="246" y="300" width="166" height="41" rx="0" fill="#eaf2ff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="329.0" y="328" font-size="19" font-weight="700" text-anchor="middle"
fill="#2463b4">id ← N0</text>
+<rect x="412" y="300" width="166" height="41" rx="0" fill="#fff2d7"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="495.0" y="328" font-size="19" font-weight="700" text-anchor="middle"
fill="#9a5200">b ← N1</text>
+<rect x="578" y="300" width="166" height="41" rx="0" fill="#e5f5ef"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="661.0" y="328" font-size="19" font-weight="700" text-anchor="middle"
fill="#10705d">c ← N2</text>
+<rect x="136" y="341" width="110" height="43" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="191.0" y="369" font-size="21" font-weight="400" text-anchor="middle"
fill="#526277">0</text>
+<rect x="246" y="341" width="166" height="43" rx="0" fill="#eaf2ff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="329.0" y="369" font-size="21" font-weight="700" text-anchor="middle"
fill="#2463b4">10</text>
+<rect x="412" y="341" width="166" height="43" rx="0" fill="#fff2d7"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="495.0" y="369" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">1</text>
+<rect x="578" y="341" width="166" height="43" rx="0" fill="#e5f5ef"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="661.0" y="369" font-size="21" font-weight="700" text-anchor="middle"
fill="#10705d">101</text>
+<rect x="136" y="384" width="110" height="43" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="191.0" y="412" font-size="21" font-weight="400" text-anchor="middle"
fill="#526277">1</text>
+<rect x="246" y="384" width="166" height="43" rx="0" fill="#eaf2ff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="329.0" y="412" font-size="21" font-weight="700" text-anchor="middle"
fill="#2463b4">20</text>
+<rect x="412" y="384" width="166" height="43" rx="0" fill="#fff2d7"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="495.0" y="412" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">22</text>
+<rect x="578" y="384" width="166" height="43" rx="0" fill="#e5f5ef"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="661.0" y="412" font-size="21" font-weight="700" text-anchor="middle"
fill="#10705d">200</text>
+<rect x="136" y="427" width="110" height="43" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="191.0" y="455" font-size="21" font-weight="400" text-anchor="middle"
fill="#526277">2</text>
+<rect x="246" y="427" width="166" height="43" rx="0" fill="#eaf2ff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="329.0" y="455" font-size="21" font-weight="700" text-anchor="middle"
fill="#2463b4">30</text>
+<rect x="412" y="427" width="166" height="43" rx="0" fill="#fff2d7"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="495.0" y="455" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">3</text>
+<rect x="578" y="427" width="166" height="43" rx="0" fill="#e5f5ef"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="661.0" y="455" font-size="21" font-weight="700" text-anchor="middle"
fill="#10705d">300</text>
+<rect x="28" y="496" width="844" height="45" rx="8" fill="#eaf2ff"
stroke="#eaf2ff" stroke-width="1.5"/>
+<text x="44" y="523" font-size="18" font-weight="400" text-anchor="start"
fill="#2463b4">Merge by row position: N2 does not replace b, because N2
contains only c.</text>
+</g>
+</svg>
diff --git a/docs/static/img/data-evolution-file-layout.svg
b/docs/static/img/data-evolution-file-layout.svg
new file mode 100644
index 0000000000..b096e30999
--- /dev/null
+++ b/docs/static/img/data-evolution-file-layout.svg
@@ -0,0 +1,79 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="600" viewBox="0 0
900 600" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">One logical range, several column files</title>
+<desc id="desc">Two adjacent file groups. Group A has two normal files over
row IDs 0 to 5, BLOB files over 0 to 1 and 2 to 5, and vector files over 0 to 2
and 3 to 5. Group B covers row IDs 6 to 9. Every dedicated file stays inside
one normal range.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="898" height="598" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">One logical range, several column files</text>
+<text x="28" y="75" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">All ranges are inclusive. Each bar represents one physical
file.</text>
+<text x="268" y="114" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Row ID</text>
+<text x="311.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">0</text>
+<text x="365.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">1</text>
+<text x="419.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">2</text>
+<text x="473.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">3</text>
+<text x="527.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">4</text>
+<text x="581.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">5</text>
+<text x="635.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">6</text>
+<text x="689.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">7</text>
+<text x="743.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">8</text>
+<text x="797.0" y="114" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">9</text>
+<line x1="284" y1="124" x2="284" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="338" y1="124" x2="338" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="392" y1="124" x2="392" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="446" y1="124" x2="446" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="500" y1="124" x2="500" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="554" y1="124" x2="554" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="608" y1="124" x2="608" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="662" y1="124" x2="662" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="716" y1="124" x2="716" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="770" y1="124" x2="770" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="824" y1="124" x2="824" y2="507" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<rect x="284" y="126" width="324" height="380" rx="8" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5" stroke-dasharray="5 5"/>
+<rect x="608" y="126" width="216" height="380" rx="8" fill="#f5f7fb"
stroke="#526277" stroke-width="1.5" stroke-dasharray="5 5"/>
+<text x="446" y="151" font-size="19" font-weight="700" text-anchor="middle"
fill="#2463b4">Group A · [0, 5]</text>
+<text x="716" y="151" font-size="19" font-weight="700" text-anchor="middle"
fill="#172b4d">Group B · [6, 9]</text>
+<text x="28" y="200" font-size="19" font-weight="700" text-anchor="start"
fill="#172b4d">Normal · id, b, c</text>
+<text x="28" y="257" font-size="19" font-weight="700" text-anchor="start"
fill="#172b4d">Normal · b update</text>
+<text x="28" y="343" font-size="19" font-weight="700" text-anchor="start"
fill="#172b4d">BLOB · image</text>
+<text x="28" y="429" font-size="19" font-weight="700" text-anchor="start"
fill="#172b4d">Vector · embedding</text>
+<rect x="286" y="176" width="320" height="35" rx="5" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="446.0" y="200.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#2463b4">N0 [0, 5]</text>
+<rect x="610" y="176" width="212" height="35" rx="5" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="716.0" y="200.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#2463b4">N2 [6, 9]</text>
+<rect x="286" y="233" width="320" height="35" rx="5" fill="#fff2d7"
stroke="#9a5200" stroke-width="1.5"/>
+<text x="446.0" y="257.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#9a5200">N1 [0, 5]</text>
+<rect x="286" y="319" width="104" height="35" rx="5" fill="#f1eafa"
stroke="#7140ad" stroke-width="1.5"/>
+<text x="338.0" y="343.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#7140ad">B0 [0, 1]</text>
+<rect x="394" y="319" width="212" height="35" rx="5" fill="#f1eafa"
stroke="#7140ad" stroke-width="1.5"/>
+<text x="500.0" y="343.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#7140ad">B1 [2, 5]</text>
+<rect x="610" y="319" width="212" height="35" rx="5" fill="#f1eafa"
stroke="#7140ad" stroke-width="1.5"/>
+<text x="716.0" y="343.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#7140ad">B2 [6, 9]</text>
+<rect x="286" y="405" width="158" height="35" rx="5" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="365.0" y="429.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#10705d">V0 [0, 2]</text>
+<rect x="448" y="405" width="158" height="35" rx="5" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="527.0" y="429.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#10705d">V1 [3, 5]</text>
+<rect x="610" y="405" width="212" height="35" rx="5" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="716.0" y="429.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#10705d">V2 [6, 9]</text>
+<text x="446" y="480" font-size="18" font-weight="400" text-anchor="middle"
fill="#2463b4">Same rows, different columns</text>
+<rect x="28" y="529" width="844" height="45" rx="8" fill="#eaf2ff"
stroke="#eaf2ff" stroke-width="1.5"/>
+<text x="44" y="556" font-size="18" font-weight="400" text-anchor="start"
fill="#2463b4">Normal versions share the full range. Dedicated files may have
smaller boundaries.</text>
+</g>
+</svg>
diff --git a/docs/static/img/data-evolution-maintenance.svg
b/docs/static/img/data-evolution-maintenance.svg
new file mode 100644
index 0000000000..fd8534c2b2
--- /dev/null
+++ b/docs/static/img/data-evolution-maintenance.svg
@@ -0,0 +1,92 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="661" viewBox="0 0
900 661" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">Compaction and deletion materialization do different
work</title>
+<desc id="desc">Starting with a normal range 0 to 3 and row 1 logically
deleted, ordinary compaction keeps all four positions and the deletion vector.
Materialization removes the deleted position and rewrites the surviving rows
with illustrative new row IDs 100, 101, and 102. Historical snapshots can
retain replaced files.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="898" height="659" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">Compaction and deletion materialization do different work</text>
+<text x="28" y="76" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Example: row ID 1 is deleted. The latest table has 3 visible
rows.</text>
+<text x="153" y="129" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Row ID</text>
+<text x="153" y="171" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Value</text>
+<rect x="165" y="106" width="136" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="233.0" y="131" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">0</text>
+<rect x="165" y="147" width="136" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="233.0" y="173" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">A</text>
+<rect x="305" y="106" width="136" height="36" rx="4" fill="#fff0f1"
stroke="#b42332" stroke-width="1.5"/>
+<text x="373.0" y="131" font-size="20" font-weight="700" text-anchor="middle"
fill="#b42332">1</text>
+<rect x="305" y="147" width="136" height="39" rx="4" fill="#fff0f1"
stroke="#b42332" stroke-width="1.5"/>
+<text x="373.0" y="173" font-size="18" font-weight="700" text-anchor="middle"
fill="#b42332">deleted</text>
+<line x1="319" y1="124" x2="427" y2="124" stroke="#b42332" stroke-width="2"/>
+<rect x="445" y="106" width="136" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="513.0" y="131" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">2</text>
+<rect x="445" y="147" width="136" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="513.0" y="173" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">C</text>
+<rect x="585" y="106" width="136" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="653.0" y="131" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">3</text>
+<rect x="585" y="147" width="136" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="653.0" y="173" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">D</text>
+<text x="445" y="213" font-size="19" font-weight="400" text-anchor="middle"
fill="#526277">Current normal range [0, 3] + deletion vector {1}</text>
+<rect x="28" y="246" width="844" height="155" rx="8" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="46" y="279" font-size="21" font-weight="700" text-anchor="start"
fill="#2463b4">Ordinary compaction</text>
+<text x="46" y="308" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Consolidate files.</text>
+<text x="46" y="337" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Keep IDs and deletion vector.</text>
+<text x="419" y="291" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Row ID</text>
+<text x="419" y="333" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Value</text>
+<rect x="431" y="268" width="98" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="480.0" y="293" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">0</text>
+<rect x="431" y="309" width="98" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="480.0" y="335" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">A</text>
+<rect x="533" y="268" width="98" height="36" rx="4" fill="#fff0f1"
stroke="#b42332" stroke-width="1.5"/>
+<text x="582.0" y="293" font-size="20" font-weight="700" text-anchor="middle"
fill="#b42332">1</text>
+<rect x="533" y="309" width="98" height="39" rx="4" fill="#fff0f1"
stroke="#b42332" stroke-width="1.5"/>
+<text x="582.0" y="335" font-size="18" font-weight="700" text-anchor="middle"
fill="#b42332">deleted</text>
+<line x1="547" y1="286" x2="617" y2="286" stroke="#b42332" stroke-width="2"/>
+<rect x="635" y="268" width="98" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="684.0" y="293" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">2</text>
+<rect x="635" y="309" width="98" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="684.0" y="335" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">C</text>
+<rect x="737" y="268" width="98" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="786.0" y="293" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">3</text>
+<rect x="737" y="309" width="98" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="786.0" y="335" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">D</text>
+<text x="635" y="382" font-size="19" font-weight="700" text-anchor="middle"
fill="#2463b4">Still [0, 3]; 3 visible rows</text>
+<rect x="28" y="420" width="844" height="170" rx="8" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="46" y="455" font-size="21" font-weight="700" text-anchor="start"
fill="#10705d">Materialize deletion vectors</text>
+<text x="46" y="484" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Rewrite surviving rows.</text>
+<text x="46" y="513" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Remove applied deletion vector.</text>
+<text x="476" y="465" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Row ID</text>
+<text x="476" y="507" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Value</text>
+<rect x="488" y="442" width="110" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="543.0" y="467" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">100</text>
+<rect x="488" y="483" width="110" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="543.0" y="509" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">A</text>
+<rect x="602" y="442" width="110" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="657.0" y="467" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">101</text>
+<rect x="602" y="483" width="110" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="657.0" y="509" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">C</text>
+<rect x="716" y="442" width="110" height="36" rx="4" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="771.0" y="467" font-size="20" font-weight="700" text-anchor="middle"
fill="#172b4d">102</text>
+<rect x="716" y="483" width="110" height="39" rx="4" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="771.0" y="509" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">D</text>
+<text x="645" y="556" font-size="18" font-weight="700" text-anchor="middle"
fill="#10705d">New IDs shown as [100, 102]</text>
+<text x="450" y="627" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">Replaced files can remain in historical snapshots until
expiration.</text>
+</g>
+</svg>
diff --git a/docs/static/img/data-evolution-partial-update.svg
b/docs/static/img/data-evolution-partial-update.svg
new file mode 100644
index 0000000000..f2e82972a9
--- /dev/null
+++ b/docs/static/img/data-evolution-partial-update.svg
@@ -0,0 +1,90 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="490" viewBox="0 0
900 490" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">Update one row; write one column over the full range</title>
+<desc id="desc">An UPDATE sets b to 22 for id 20. The original file N0
contains row IDs 0, 1, and 2. New file N1 contains b values 1, 22, and 3 over
the same range. Values 1 and 3 are preserved. N0 remains unchanged and
continues to supply id and c.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="898" height="488" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">Update one row; write one column over the full range</text>
+<rect x="28" y="68" width="844" height="43" rx="6" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="44" y="96" font-size="20" font-weight="700" text-anchor="start"
fill="#172b4d">UPDATE t SET b = 22 WHERE id = 20</text>
+<text x="28" y="150" font-size="21" font-weight="700" text-anchor="start"
fill="#2463b4">N0 · existing file [0, 2]</text>
+<text x="559" y="150" font-size="21" font-weight="700" text-anchor="start"
fill="#9a5200">N1 · new file [0, 2]</text>
+<rect x="28" y="170" width="89" height="40" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.5" y="197" font-size="18" font-weight="700" text-anchor="middle"
fill="#526277">Row ID</text>
+<rect x="117" y="170" width="110" height="40" rx="0" fill="#eaf2ff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="172.0" y="197" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">id</text>
+<rect x="227" y="170" width="110" height="40" rx="0" fill="#eaf2ff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="282.0" y="197" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">b</text>
+<rect x="337" y="170" width="110" height="40" rx="0" fill="#eaf2ff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="392.0" y="197" font-size="18" font-weight="700" text-anchor="middle"
fill="#2463b4">c</text>
+<rect x="28" y="210" width="89" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.5" y="240" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">0</text>
+<rect x="117" y="210" width="110" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="172.0" y="240" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">10</text>
+<rect x="227" y="210" width="110" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="282.0" y="240" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">1</text>
+<rect x="337" y="210" width="110" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="392.0" y="240" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">100</text>
+<rect x="28" y="257" width="89" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.5" y="287" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">1</text>
+<rect x="117" y="257" width="110" height="47" rx="0" fill="#fff2d7"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="172.0" y="287" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">20</text>
+<rect x="227" y="257" width="110" height="47" rx="0" fill="#fff2d7"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="282.0" y="287" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">2</text>
+<rect x="337" y="257" width="110" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="392.0" y="287" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">200</text>
+<rect x="28" y="304" width="89" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="72.5" y="334" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">2</text>
+<rect x="117" y="304" width="110" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="172.0" y="334" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">30</text>
+<rect x="227" y="304" width="110" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="282.0" y="334" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">3</text>
+<rect x="337" y="304" width="110" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="392.0" y="334" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">300</text>
+<path d="M 451 233 L 550 233" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<path d="M 451 280 L 550 280" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<path d="M 451 327 L 550 327" fill="none" stroke="#526277" stroke-width="2"
marker-end="url(#arrow)"/>
+<rect x="559" y="170" width="89" height="40" rx="0" fill="#f5f7fb"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="603.5" y="197" font-size="18" font-weight="700" text-anchor="middle"
fill="#526277">Row ID</text>
+<rect x="648" y="170" width="90" height="40" rx="0" fill="#fff2d7"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="693.0" y="197" font-size="18" font-weight="700" text-anchor="middle"
fill="#9a5200">b</text>
+<rect x="559" y="210" width="89" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="603.5" y="240" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">0</text>
+<rect x="648" y="210" width="90" height="47" rx="0" fill="#fff2d7"
stroke="#9a5200" stroke-width="1.5"/>
+<text x="693" y="240" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">1</text>
+<text x="752" y="240" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">preserved</text>
+<rect x="559" y="257" width="89" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="603.5" y="287" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">1</text>
+<rect x="648" y="257" width="90" height="47" rx="0" fill="#fff2d7"
stroke="#9a5200" stroke-width="1.5"/>
+<text x="693" y="287" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">22</text>
+<text x="752" y="287" font-size="18" font-weight="400" text-anchor="start"
fill="#9a5200">changed</text>
+<rect x="559" y="304" width="89" height="47" rx="0" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="603.5" y="334" font-size="21" font-weight="400" text-anchor="middle"
fill="#172b4d">2</text>
+<rect x="648" y="304" width="90" height="47" rx="0" fill="#fff2d7"
stroke="#9a5200" stroke-width="1.5"/>
+<text x="693" y="334" font-size="21" font-weight="700" text-anchor="middle"
fill="#9a5200">3</text>
+<text x="752" y="334" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">preserved</text>
+<text x="28" y="383" font-size="19" font-weight="700" text-anchor="start"
fill="#2463b4">N0 stays unchanged.</text>
+<text x="559" y="383" font-size="18" font-weight="700" text-anchor="start"
fill="#9a5200">Same 3 positions. Only b is written.</text>
+<rect x="28" y="411" width="844" height="70" rx="8" fill="#eaf2ff"
stroke="#eaf2ff" stroke-width="1.5"/>
+<text x="44" y="438" font-size="18" font-weight="400" text-anchor="start"
fill="#2463b4">The predicate selects row 1, but N1 must still cover every row
ID in [0, 2].</text>
+<text x="44" y="463" font-size="18" font-weight="400" text-anchor="start"
fill="#2463b4">Columns id and c continue to come from N0.</text>
+</g>
+</svg>
diff --git a/docs/static/img/data-evolution-range-contract.svg
b/docs/static/img/data-evolution-range-contract.svg
new file mode 100644
index 0000000000..65fafd2225
--- /dev/null
+++ b/docs/static/img/data-evolution-range-contract.svg
@@ -0,0 +1,82 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="900" height="642" viewBox="0 0
900 642" role="img" aria-labelledby="title desc">
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<title id="title">Containment has a direction</title>
+<desc id="desc">Each candidate is checked separately against normal ranges 0
to 5 and 6 to 9. Normal 0 to 5 is valid; normal 0 to 3 is invalid. Dedicated 2
to 4 and 0 to 5 are valid. Dedicated 4 to 7 and 0 to 9 are invalid because they
span normal ranges.</desc>
+<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7"
markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z"
fill="#526277"/></marker></defs>
+<g font-family="Arial, Helvetica, sans-serif" fill="#172b4d">
+<rect x="1" y="1" width="898" height="640" rx="12" fill="#ffffff"
stroke="#d7dfeb" stroke-width="1.5"/>
+<text x="28" y="43" font-size="27" font-weight="700" text-anchor="start"
fill="#172b4d">Containment has a direction</text>
+<text x="28" y="75" font-size="18" font-weight="400" text-anchor="start"
fill="#526277">Each candidate below is a separate alternative against the same
normal ranges.</text>
+<text x="322" y="117" font-size="17" font-weight="400" text-anchor="end"
fill="#526277">Row ID</text>
+<text x="360.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">0</text>
+<text x="405.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">1</text>
+<text x="450.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">2</text>
+<text x="495.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">3</text>
+<text x="540.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">4</text>
+<text x="585.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">5</text>
+<text x="630.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">6</text>
+<text x="675.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">7</text>
+<text x="720.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">8</text>
+<text x="765.5" y="117" font-size="18" font-weight="400" text-anchor="middle"
fill="#526277">9</text>
+<line x1="338" y1="127" x2="338" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="383" y1="127" x2="383" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="428" y1="127" x2="428" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="473" y1="127" x2="473" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="518" y1="127" x2="518" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="563" y1="127" x2="563" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="608" y1="127" x2="608" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="653" y1="127" x2="653" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="698" y1="127" x2="698" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="743" y1="127" x2="743" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<line x1="788" y1="127" x2="788" y2="548" stroke="#d7dfeb" stroke-width="1.5"
stroke-dasharray="3 5"/>
+<rect x="340" y="132" width="266" height="35" rx="5" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="473.0" y="156.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#2463b4">Normal [0, 5]</text>
+<rect x="610" y="132" width="176" height="35" rx="5" fill="#eaf2ff"
stroke="#2463b4" stroke-width="1.5"/>
+<text x="698.0" y="156.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#2463b4">Normal [6, 9]</text>
+<line x1="608" y1="126" x2="608" y2="552" stroke="#b42332" stroke-width="2"
stroke-dasharray="6 5"/>
+<text x="28" y="157" font-size="19" font-weight="700" text-anchor="start"
fill="#172b4d">Existing normal files</text>
+<text x="28" y="227" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d">Normal: full range</text>
+<rect x="340" y="204" width="266" height="35" rx="5" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="473.0" y="228.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#10705d">[0, 5]</text>
+<text x="804" y="227" font-size="16" font-weight="700" text-anchor="start"
fill="#10705d">VALID</text>
+<text x="28" y="287" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d">Normal: only a subset</text>
+<rect x="340" y="264" width="176" height="35" rx="5" fill="#fff0f1"
stroke="#b42332" stroke-width="1.5"/>
+<text x="428.0" y="288.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#b42332">[0, 3]</text>
+<text x="804" y="287" font-size="16" font-weight="700" text-anchor="start"
fill="#b42332">INVALID</text>
+<text x="28" y="347" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d">Dedicated: contained</text>
+<rect x="430" y="324" width="131" height="35" rx="5" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="495.5" y="348.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#10705d">[2, 4]</text>
+<text x="804" y="347" font-size="16" font-weight="700" text-anchor="start"
fill="#10705d">VALID</text>
+<text x="28" y="407" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d">Dedicated: equal</text>
+<rect x="340" y="384" width="266" height="35" rx="5" fill="#e5f5ef"
stroke="#10705d" stroke-width="1.5"/>
+<text x="473.0" y="408.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#10705d">[0, 5]</text>
+<text x="804" y="407" font-size="16" font-weight="700" text-anchor="start"
fill="#10705d">VALID</text>
+<text x="28" y="467" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d">Dedicated: crosses boundary</text>
+<rect x="520" y="444" width="176" height="35" rx="5" fill="#fff0f1"
stroke="#b42332" stroke-width="1.5"/>
+<text x="608.0" y="468.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#b42332">[4, 7]</text>
+<text x="804" y="467" font-size="16" font-weight="700" text-anchor="start"
fill="#b42332">INVALID</text>
+<text x="28" y="527" font-size="18" font-weight="700" text-anchor="start"
fill="#172b4d">Dedicated: covers both</text>
+<rect x="340" y="504" width="446" height="35" rx="5" fill="#fff0f1"
stroke="#b42332" stroke-width="1.5"/>
+<text x="563.0" y="528.5" font-size="18" font-weight="700"
text-anchor="middle" fill="#b42332">[0, 9]</text>
+<text x="804" y="527" font-size="16" font-weight="700" text-anchor="start"
fill="#b42332">INVALID</text>
+<rect x="28" y="577" width="844" height="45" rx="8" fill="#fff0f1"
stroke="#fff0f1" stroke-width="1.5"/>
+<text x="44" y="604" font-size="18" font-weight="400" text-anchor="start"
fill="#b42332">A dedicated range must fit inside ONE normal range, not the
union of two.</text>
+</g>
+</svg>
