zy-kkk opened a new pull request, #68453:
URL: https://github.com/apache/doris/pull/68453
### What problem does this PR solve?" 起的内容（按根 `AGENTS.md` 模板）。
示例输出全部是 2026-09-22 在本地 4.1 实例上用 mysql client 跑出来的原样复制，
只把本机 MinIO / REST 桩地址保留为 127.0.0.1（回归环境里就是这两个服务）。
---
### What problem does this PR solve?
Issue Number: close #66491
Related PR: #65730 (baseline Lance read integration, which already carried
`FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation)
Problem Summary:
Lance catalogs could select a dataset version on the FE, but nothing proved
it end to end, the error messages leaked SDK internals (`NumberFormatException:
For input string`, storage paths and Rust source locations), and tables whose
versions are managed by a REST Namespace (`DescribeTable` returns
`managed_versioning = true`) were rejected outright with "not supported by the
current BE reader".
This PR completes the four scope items of the issue:
1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF
'<timestamp>'` are covered by regression suites on a filesystem catalog and a
REST catalog, with a committed three-version fixture (`time_travel.lance`) and
the existing `multi_frag.lance` (three appends, then a delete) so that reading
an older version across a deletion file is exercised on the BE.
The semantics follow the Iceberg and Paimon integrations: the selected
version is the latest one whose commit time is not later than the timestamp,
parsed in the session time zone with second or millisecond precision and
compared with commit times at millisecond precision (the precision a namespace
reports them in, so storage and namespace resolution agree); a timestamp
earlier than the first version is an error.
2. **REST Namespace managed versioning** – the FE now opens such a table
through the SDK's namespace client
(`Dataset.open().namespaceClient(..).tableId(..)`), so the latest version comes
from `ListTableVersions`, an explicit version from `DescribeTableVersion`, and
`FOR TIME AS OF` from the commit times (`timestamp_millis`) the namespace
reports. The SDK finalizes a still-staged manifest to its canonical
`_versions/` path while opening, so the BE keeps opening the dataset by URI and
version number through lance-c exactly as before; no BE or thrift change is
needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables.
3. **Selected version fixed for the whole statement** – schema binding,
fragment planning, predicate pushdown, metadata COUNT and the BE scan all use
the selected version; two references to the same table in one statement can
select different versions (self join / union tests).
4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch
and version it points at, read from the dataset's `_refs/tags/` for both
storage-versioned and managed tables (Lance's namespace tag APIs expose the
same refs; the selected version of a managed table is then resolved through the
namespace); a tag created on a branch selects that branch's version, not the
same number on `main`. As for Iceberg and Paimon, a non-numeric `FOR VERSION AS
OF 'v2'` names a tag. `@branch(main)` is `main`; `@tag` / `@branch` take
exactly one name and reject extra parameters. Inside a branch, versions and
times select among the branch's own versions, which start at the version it was
created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`,
optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the
BE receives the branch directory as the dataset URI and needs no change (a
branch is a shallow clone with its own `_versions/`). For managed table
s the SDK derives the branch from that directory and sends it in the `branch`
field of the version APIs. Previously `@tag` / `@branch` were silently ignored
and the query read the latest version of `main`.
5. **Errors** – a missing version (never committed, or removed by
`cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')]
was not found[ in the namespace]` without the storage path; a missing tag or
branch, an out-of-range or non-positive version (a signed number is a version,
not a tag name), a timestamp before the first version and an unparsable
timestamp have their own messages; a namespace that lists no versions is
reported as such; a table that is only declared in the namespace fails with
"declared in the namespace but has no data yet"; for a managed table, a
namespace `location` that is empty or differs from `table_uri` fails the query
before anything is read; every other failure keeps the sanitized provider
message.
Design decisions worth calling out:
- For a managed table the namespace is the source of truth. A version whose
manifest is still in storage but which the namespace does not record is
reported as not found, and the table's latest version is the namespace's latest
even when storage already holds a newer manifest. Reading unrecorded manifests
would expose staged or rolled-back commits.
- `timestamp_millis` is optional in the Lance Namespace spec. If a namespace
lists its versions without it, the only commit times available are the ones in
the storage manifests, so `FOR TIME AS OF` uses those, restricted to the
versions the namespace lists so that the selected version is always one the
namespace records. A manifest records when the writer committed the data, which
can be earlier than when the namespace registered the version, so the selection
follows the writer's clock in that case.
- The SDK opens and checks out a namespace-managed dataset with its own
native namespace client (the JNI takes the native handle of a REST or Directory
namespace instead of calling back into the Java object), so those operations
are not serialized on the FE's namespace lock; only the FE's own namespace
requests are.
- `FOR VERSION AS OF` on a dataset whose directory is missing altogether
reports "version N was not found", because the SDK probes the version's
manifest path first; the provider's own message is kept in the exception cause,
handled the same way as every other Lance error.
- `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its
`lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace
client, whose `branch` field on the version requests is what managed-table
branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled
unchanged and every Lance unit test class and regression suite passes on it.
The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and
never writes a dataset, so the newer FE release does not change what the BE can
read.
- An expired version (removed by cleanup) is indistinguishable from one that
never existed, on storage and in a namespace, so both get the same message. "in
the namespace" is only added when the namespace itself reported the miss.
- Every selector other than an explicit version on `main` is a checkout from
one open of the latest `main`, and a managed `FOR TIME AS OF` whose namespace
reports commit times is resolved before anything is opened. The namespace
version list is fetched completely, once per read, without a page size: Lance's
Directory namespace applies a limit without returning a page token, and neither
the order nor monotonic commit times can be relied on to stop early.
- Reading a managed table whose manifest is still staged makes the SDK copy
it to its canonical path, so read credentials must allow that write; an
access-denied failure on a managed table says so.
- The BE reads by URI and version number, which assumes the manifest a
namespace records ends up at its canonical path `_versions/<u64::MAX -
v>.manifest`. A staged manifest from a normal commit is moved there on read; a
namespace that keeps finalized manifests elsewhere cannot be detected by the
FE, and the BE then either fails to find the version or reads a different
manifest with the same number at the canonical path. The documentation lists
this as a limitation.
- Known limitations: two references to the same branch in one statement
resolve the branch head independently, and the optimizer's row count is that of
the latest `main` (both shared with the Iceberg integration).
`vector_search()`, `full_text_search()` and index inspection use the latest
`main` and reject a version, tag or branch in their `table` argument. Index
paths open by URI and are only reachable for filesystem catalogs, whose
Directory namespace never manages versions; an assertion guards that.
- A Lance branch is a shallow clone whose manifests record the parent's
location as an absolute URI (`Manifest.base_paths`), so a branch is readable
only where it was created. The committed `time_travel.lance` branch was
therefore created against the fixture's final location
`s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py
--create-branch`) and synced back; the fixture's self-check verifies that
reference.
Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions
1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3
/ 4..6 / 7..9):
```sql
mysql> SET enable_file_scanner_v2 = true;
mysql> SET time_zone = 'UTC';
mysql> CREATE CATALOG lance_demo_fs PROPERTIES (
-> "type" = "lance",
-> "lance.catalog.type" = "filesystem",
-> "warehouse" = "s3://warehouse/lance",
-> "s3.endpoint" = "http://127.0.0.1:19000";,
-> "s3.access_key" = "admin",
-> "s3.secret_key" = "password",
-> "s3.region" = "us-east-1",
-> "use_path_style" = "true"
-> );
mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1
ORDER BY row_id;
+--------+------+
| row_id | tag |
+--------+------+
| 1 | v1 |
| 2 | v1 |
| 3 | v1 |
+--------+------+
mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel
FOR TIME AS OF '2026-09-19 13:06:10';
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3
-> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a
-> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON
a.row_id = b.row_id
-> ORDER BY a.row_id;
+--------+--------+--------+
| row_id | tag_v1 | tag_v3 |
+--------+--------+--------+
| 1 | v1 | v1 |
| 2 | v1 | v1 |
| 3 | v1 | v1 |
+--------+--------+--------+
mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION
AS OF 99;
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of
default.time_travel was not found
mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS
OF '2026-09-19 13:06:07';
ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to load Lance table
metadata for default.time_travel: IllegalArgumentException: Lance dataset has
no version at or before '2026-09-19 13:06:07'
```
A REST Namespace with managed versioning. `time_travel_managed` records
versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage
already holds version 3, and `time_travel_managed_partial` records 1 and 3:
```sql
mysql> CREATE CATALOG lance_demo_rest PROPERTIES (
-> "type" = "lance",
-> "lance.catalog.type" = "rest",
-> "lance.rest.uri" = "http://127.0.0.1:19102";,
-> "lance.rest.security.type" = "bearer",
-> "lance.rest.bearer-token" = "doris-lance-rest-test-token",
-> "lance.namespace.root_database" = "default",
-> "s3.endpoint" = "http://127.0.0.1:19000";,
-> "s3.region" = "us-east-1",
-> "use_path_style" = "true"
-> );
mysql> SELECT count(*), max(row_id) FROM
lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2;
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT count(*), max(row_id) FROM
lance_demo_rest.`default`.time_travel_managed_lagging;
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT count(*) FROM
lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3;
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of
default.time_travel_managed_lagging was not found in the namespace
mysql> SELECT count(*), max(row_id) FROM
lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF
'2026-09-19 13:06:10';
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 3 | 3 |
+----------+-------------+
mysql> EXPLAIN SELECT row_id FROM
lance_demo_rest.`default`.time_travel_managed_lagging;
...
0:VLANCE_SCAN_NODE(44)
table: lance_demo_rest.default.time_travel_managed_lagging
lanceCatalogType=rest
lanceVersion=2
lanceManagedVersioning=true
lanceFragments=2
lanceFragmentGrouping=FRAGMENT
```
Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch
`dev` forked from version 2 with one extra row; on
`time_travel_managed_partial` the tag `v2` points at the version the namespace
no longer records):
```sql
mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY
row_id;
+--------+------+
| row_id | tag |
+--------+------+
| 1 | v1 |
| 2 | v1 |
| 3 | v1 |
| 4 | v2 |
| 5 | v2 |
| 6 | v2 |
+--------+------+
mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER
BY row_id;
+--------+------+
| row_id | tag |
+--------+------+
| 1 | v1 |
| 2 | v1 |
| 3 | v1 |
| 4 | v2 |
| 5 | v2 |
| 6 | v2 |
| 100 | dev |
+--------+------+
mysql> SELECT count(*), max(row_id) FROM
lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2;
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT count(*) FROM
lance_demo_fs.`default`.time_travel@tag(no_such_tag);
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of
default.time_travel was not found
mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope);
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of
default.time_travel was not found
mysql> SELECT count(*), max(row_id) FROM
lance_demo_rest.`default`.time_travel_managed@branch(dev);
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 7 | 100 |
+----------+-------------+
mysql> SELECT count(*), max(row_id) FROM
lance_demo_rest.`default`.time_travel_managed@tag(v1);
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 3 | 3 |
+----------+-------------+
mysql> SELECT count(*) FROM
lance_demo_rest.`default`.time_travel_managed_partial@tag(v2);
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of
default.time_travel_managed_partial (tag 'v2') was not found in the namespace
mysql> EXPLAIN SELECT row_id FROM
lance_demo_fs.`default`.time_travel@branch(dev);
...
0:VLANCE_SCAN_NODE(44)
table: lance_demo_fs.default.time_travel
lanceCatalogType=filesystem
lanceVersion=3
lanceManagedVersioning=false
lanceBranch=dev
lanceFragments=3
lanceFragmentGrouping=FRAGMENT
```
Test environment changes: the Lance REST fixture (`lance_rest_server.py`)
gains `ListTableVersions` / `DescribeTableVersion` for tables listed in
`LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed
three-version dataset with tags and a branch, carried over as-is by
`lance_build_preinstalled_catalog.py` because the suites hard-code its commit
times (`lance_build_time_travel.py` regenerates it and prints the times to
update; `--create-branch` forks the branch at the uploaded location). The stub
additionally serves the `branch` parameter of the version APIs.
### Release note
Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)`
and `@branch(name)`, including tables whose versions are managed by a REST
Namespace (`managed_versioning = true`). Missing versions, tags and branches
and invalid selectors now report clear errors.
### Check List (For Author)
- Test: Regression test / Unit Test
- `external_table_p0/lance/test_lance_time_travel`,
`test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture +
MinIO)
- `LanceManagedVersioningTest` (real SDK dataset behind a REST stub:
managed / partial / lagging / staged manifest / untimed / growing / server
error / location checks / declared-only / cleaned-up version / tags / branches
/ paging / empty version list), plus `test_lance_vector_search` and
`test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest`,
`LanceTableAccessCacheTest`, `LanceRestCatalogTest`, `LanceScanNodeTest`,
`LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest`
- Behavior changed: Yes. Tables with `managed_versioning = true` were
rejected and are now readable; error messages for invalid or missing versions
changed to the user-facing form above; `@tag` / `@branch` on a Lance table now
select the tag or branch (including a tag that points into a branch) instead of
silently reading the latest version of `main`; FE dependency `lance-core`
11.0.0 → 12.0.0.
- Does this need documentation: Yes (doris-website PR to follow: feature
table, REST catalog caution, new "Time Travel" section with the
managed-versioning limitations)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
