This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new a82a5e4e11f [doc](lance) delete master branch doc about lance catalog
(#4086)
a82a5e4e11f is described below
commit a82a5e4e11fecda5b9fe099b209f4cf3ce9a2fc4
Author: zhangstar333 <[email protected]>
AuthorDate: Tue Aug 25 11:44:48 2026 +0800
[doc](lance) delete master branch doc about lance catalog (#4086)
## Versions
- [x] dev
- [ ] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
- [ ] Updated required version and language counterparts, or explained
why not
- [ ] If only one language changed, confirmed whether source/translation
counterparts need sync
---
docs/lakehouse/catalogs/lance-catalog.mdx | 541 ---------------------
.../current/lakehouse/catalogs/lance-catalog.mdx | 541 ---------------------
sidebars.ts | 1 -
3 files changed, 1083 deletions(-)
diff --git a/docs/lakehouse/catalogs/lance-catalog.mdx
b/docs/lakehouse/catalogs/lance-catalog.mdx
deleted file mode 100644
index cf60bedd4e5..00000000000
--- a/docs/lakehouse/catalogs/lance-catalog.mdx
+++ /dev/null
@@ -1,541 +0,0 @@
----
-{
- "title": "Lance Catalog",
- "language": "en",
- "description": "Apache Doris Lance Catalog guide: access Lance datasets
through Filesystem or REST Namespace catalogs, with parallel reads, predicate
pushdown, S3/Local TVFs, and vector search."
-}
----
-
-:::note
-This is an experimental feature.
-
-Lance Catalog is supported starting from Apache Doris 4.2.
-:::
-
-Lance is a columnar data format designed for analytics and AI workloads. Doris
can use a Lance Catalog to discover databases and tables in a Lance Namespace
and directly query Lance datasets stored on a local file system or
S3-compatible object storage.
-
-Doris currently provides read-only access to Lance. Creating, writing,
updating, or deleting Lance tables is not supported.
-
-## Feature Overview
-
-| Feature | Support |
-|---|---|
-| Filesystem Catalog | Supports warehouses on a local file system, `file://`,
or `s3://` |
-| REST Catalog | Supports Lance REST Namespace with no authentication, Bearer
Token, API Key, or custom HTTP headers |
-| Metadata access | Supports `SHOW DATABASES`, `SHOW TABLES`, and `DESC` |
-| Data queries | Supports column pruning, parallel Lance Fragment scans, and
snapshot-consistent reads of the current version |
-| Predicate pushdown | Supports pushing compatible scalar predicates down to
Lance |
-| File TVFs | Supports querying Lance datasets directly through `s3()` and
`local()` |
-| Vector search | Uses physical Lance index segments as parallel splits, keeps
uncovered Fragments as Flat Search splits, and performs a Doris global Top-K
merge |
-| Writing to Lance | Not supported |
-| Time Travel | Not supported |
-| Full-Text Search / Hybrid Search | Not supported |
-
-## Lance Version and Compatibility
-
-The Doris BE data reader is built with `lance-c v0.1.6`. In Doris, this
version is bound to Lance `9.1.0-beta.3` at Lance commit `e934cc2c`. The
`lance-c` and Lance Rust crate versions identify the reader implementation
integrated with Doris. They are different from the Lance `data_storage_version`
recorded in a dataset.
-
-The following table describes the file-format compatibility of this reader:
-
-| `data_storage_version` | Read support | Notes |
-|---|---|---|
-| `0.1` / `legacy` | Supported | Original Lance file format. |
-| `2.0` (writer-option alias `0.3`) | Supported | An earlier version of the
Lance v2 file format. |
-| `2.1` / `stable` | Supported; default stable format | In the embedded Lance
version, the `stable` writer option and the default format for new datasets
both resolve to `2.1`. |
-| `2.2` | Supported | The embedded Lance version treats this as a stable
format, but it is not the default writer format. |
-| `2.3` / `next` | Experimental; not guaranteed | The embedded Lance version
marks `2.3` as unstable, and the `next` writer option resolves to `2.3`. |
-| A later or unknown version | Not supported | Opening or scanning the dataset
may fail with an unsupported storage-version error. |
-
-Lance SDK release numbers and file-format versions are independent. A dataset
written by an older or newer Lance SDK is readable only when its storage
format, required table feature flags, index format, and Arrow/Lance data types
are all understood by the versions embedded in Doris. Consequently:
-
-- Doris is expected to read datasets written with the `0.1`, `2.0`, `2.1`, and
`2.2` storage formats, subject to the type limitations documented below.
-- Forward compatibility is not guaranteed. A dataset written or modified by a
later Lance release may be unreadable if it uses a newer storage format, an
unknown required manifest feature, a newer index format, or an unsupported
extension type.
-- For datasets that must remain readable by this Doris release, use the
current default stable format, `2.1`, and do not use `next`. If a newer writer
or optional Lance feature is introduced, validate the resulting dataset with
the target Doris release before using it in production.
-
-## Configure a Catalog
-
-### Syntax
-
-```sql
-CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "<filesystem|rest>",
- {CatalogProperties},
- {StorageProperties},
- {CommonProperties}
-);
-```
-
-### Common Properties
-
-| Property | Required | Default | Description |
-|---|---|---|---|
-| `type` | Yes | - | Must be `lance`. |
-| `lance.catalog.type` | No | `filesystem` | Catalog type. Valid values are
`filesystem` and `rest`. |
-| `lance.namespace.parent` | No | Empty | Limits access to the specified Lance
Namespace and its child Namespaces. With the default delimiter, for example,
`production$analytics` represents a two-level Namespace. |
-| `lance.namespace.delimiter` | No | `$` | Delimiter used to parse
`lance.namespace.parent`. It is also passed to the REST Namespace client. This
property does not change how multilevel Namespaces are displayed in Doris. |
-| `lance.namespace.root_database` | No | `default` | Doris database name to
which the root Lance Namespace is mapped. |
-
-### Filesystem Catalog
-
-A Filesystem Catalog discovers Lance Namespaces and tables directly from a
warehouse directory.
-
-| Property | Required | Description |
-|---|---|---|
-| `warehouse` | Yes | Root path of the Lance warehouse. Local absolute paths,
`file://` URIs, and `s3://` URIs are supported. |
-
-#### Use S3-Compatible Object Storage
-
-The following example creates a Catalog for MinIO:
-
-```sql
-CREATE CATALOG lance_catalog PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "s3://my-bucket/lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true"
-);
-```
-
-When accessing AWS S3, you can omit `s3.endpoint` and configure credentials,
Region, and Path Style for your environment.
-
-#### Use a Local File System
-
-```sql
-CREATE CATALOG lance_local PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "/data/lance"
-);
-```
-
-For a local file system, `warehouse` must be an absolute path. The FE must be
able to read Namespace and table metadata through this path, and each BE that
executes a query must be able to access the data through the same path. In a
multi-node deployment, mount the same shared directory on all relevant FE and
BE nodes.
-
-### REST Catalog
-
-A REST Catalog obtains Namespaces, table locations, and storage access
parameters through Lance REST Namespace. A REST Catalog neither requires nor
permits the `warehouse` property.
-
-| Property | Required | Default | Description |
-|---|---|---|---|
-| `lance.rest.uri` | Yes | - | REST service URI. It must use `http://` or
`https://`. |
-| `lance.rest.security.type` | No | `none` | Authentication type. Valid values
are `none`, `bearer`, and `api_key`. |
-| `lance.rest.bearer-token` | Yes for Bearer authentication | - | Bearer
Token. |
-| `lance.rest.api-key` | Yes for API Key authentication | - | API Key sent in
the `x-api-key` header. |
-| `lance.rest.header.<header-name>` | No | - | Custom HTTP header sent to the
REST service. Use the dedicated authentication properties above for
authentication headers. |
-
-The following example creates a REST Catalog using a Bearer Token:
-
-```sql
-CREATE CATALOG lance_rest PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "rest",
- "lance.rest.uri" = "https://lance.example.com";,
- "lance.rest.security.type" = "bearer",
- "lance.rest.bearer-token" = "your-token"
-);
-```
-
-For API Key authentication, replace the authentication properties with:
-
-```sql
-"lance.rest.security.type" = "api_key",
-"lance.rest.api-key" = "your-api-key"
-```
-
-If the REST service returns temporary storage credentials, Doris uses those
credentials to access the corresponding Lance table. You can also configure
`s3.endpoint`, `s3.access_key`, `s3.secret_key`, `s3.region`, and
`use_path_style` in the Catalog as the default object storage access parameters.
-
-:::caution
-The current BE Reader does not support Lance tables whose versions are managed
by REST Namespace (Managed Versioning).
-:::
-
-## Namespace Mapping
-
-Lance supports multilevel Namespaces, while a Doris Catalog represents each
Namespace as a database name:
-
-| Lance Namespace | Doris Database Name |
-|---|---|
-| Root Namespace | `default`; configurable through
`lance.namespace.root_database` |
-| `doris` | `doris` |
-| `doris.analytics` | `doris.analytics` |
-
-Doris joins the levels of a multilevel Namespace with `.` to form a database
name. Use backticks when referencing a database name that contains `.`:
-
-```sql
-SHOW TABLES FROM lance_catalog.`doris.analytics`;
-
-SELECT *
-FROM lance_catalog.`doris.analytics`.user_features;
-```
-
-Use `lance.namespace.parent` to limit a Catalog to a Namespace subtree. For
example:
-
-```sql
-CREATE CATALOG lance_analytics PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "s3://my-bucket/lance",
- "lance.namespace.parent" = "production$analytics",
- "s3.region" = "us-east-1"
-);
-```
-
-Doris then displays only the tables and child Namespaces below
`production.analytics`.
-
-## Query Lance Tables
-
-After creating a Catalog, you can browse and query Lance tables in the same
way as other external tables:
-
-```sql
-SHOW DATABASES FROM lance_catalog;
-
-SHOW TABLES FROM lance_catalog.default;
-
-DESC lance_catalog.default.user_profiles;
-
-SELECT user_id, name, age
-FROM lance_catalog.default.user_profiles
-WHERE age >= 18
-ORDER BY user_id
-LIMIT 100;
-```
-
-You can also load data from Lance into a Doris internal table:
-
-```sql
-INSERT INTO internal.demo.user_profiles
-SELECT user_id, name, age
-FROM lance_catalog.default.user_profiles;
-```
-
-For a regular Catalog query, Doris pins a Lance dataset version during
planning and generates scan tasks by Fragment. A query therefore reads a
consistent snapshot, while multiple Scanners can read different Fragments in
parallel without every Scanner repeatedly scanning the entire dataset.
-
-## Type Mapping
-
-| Lance / Arrow Type | Doris Type | Description |
-|---|---|---|
-| `bool` | `BOOLEAN` | |
-| `int8` | `TINYINT` | |
-| `uint8` | `SMALLINT` | Losslessly widened unsigned integer |
-| `int16` | `SMALLINT` | |
-| `uint16` | `INT` | Losslessly widened unsigned integer |
-| `int32` | `INT` | |
-| `uint32` | `BIGINT` | Losslessly widened unsigned integer |
-| `int64` | `BIGINT` | |
-| `uint64` | `LARGEINT` | Losslessly widened unsigned integer |
-| `float16` | `FLOAT` | Widened to a 32-bit floating-point value |
-| `float32` | `FLOAT` | |
-| `float64` | `DOUBLE` | |
-| `decimal128(P,S)` | `DECIMAL(P,S)` | Maximum precision is 38 |
-| `decimal256(P,S)` | `DECIMAL(P,S)` | Maximum precision is 76 |
-| `utf8`, `large_utf8` | `TEXT` | |
-| `binary`, `large_binary` | `VARBINARY(2147483647)` | |
-| `fixed_size_binary(N)` | `VARBINARY(N)` | Preserves the fixed byte width |
-| `date32(day)`, `date64(ms)` | `DATE` | A `date64` value must represent a
complete calendar day |
-| `time32(s)` | `TIME(0)` | |
-| `time32(ms)` | `TIME(3)` | |
-| `time64(us)`, `time64(ns)` | `TIME(6)` | Nanosecond precision is truncated
to microseconds |
-| Timezone-naive `timestamp(s)` | `DATETIME` | Not converted according to the
Session Time Zone |
-| Timezone-naive `timestamp(ms)` | `DATETIME(3)` | Not converted according to
the Session Time Zone |
-| Timezone-naive `timestamp(us)`, `timestamp(ns)` | `DATETIME(6)` | Nanosecond
precision is truncated to microseconds |
-| Timezone-aware `timestamp` | `TIMESTAMPTZ(0-6)` | Preserves the instant and
displays it in the Doris Session Time Zone |
-| `struct` | `STRUCT` | Child fields are mapped recursively |
-| `list`, `large_list`, `fixed_size_list` | `ARRAY` | Element types are mapped
recursively |
-| `map` | `MAP` | Key and value types are mapped recursively |
-
-The following types are not currently supported:
-
-- Arrow `null` and `duration`.
-- Arrow/Lance Extension types with `ARROW:extension:name` metadata, including
Lance Blob v2, Arrow JSON Extension, and Lance BFloat16 Extension.
-- Complex types whose child types cannot be mapped recursively.
-- Arrow Dictionary types that preserve the Dictionary marker.
-
-For an unsupported top-level column, `DESC` on a Catalog table and `DESC
FUNCTION` on a Lance file TVF both preserve the column and display `unknown
type: UNSUPPORTED_TYPE`. If any child of a complex type cannot be mapped, the
whole top-level complex column is marked unsupported. Queries can still project
only supported columns. Doris reports an error during analysis when SQL
projects an unsupported column. For example:
-
-```sql
-SELECT * EXCEPT(blob_col, json_col)
-FROM lance_catalog.default.all_types;
-```
-
-:::note
-Some Lance Java SDK versions may lose the Dictionary marker while reading a
Schema and expose a Dictionary column as its physical index type. This behavior
does not mean that Doris supports the logical Dictionary values and must not be
relied upon.
-:::
-
-## Predicate Pushdown
-
-Doris converts semantically compatible predicates into Substrait expressions
and passes them to Lance for evaluation during reads. The Doris BE does not
evaluate a condition again after the entire condition has been pushed down.
Conditions that cannot be pushed down safely remain in Doris.
-
-### Data Types Supported for Pushdown
-
-| Lance / Arrow Type | Pushdown Support |
-|---|---|
-| `bool` | Equality, null checks, and logical operations; ordering comparisons
are not supported |
-| `int8/16/32/64` | Supported |
-| `uint8/16/32/64` | Supported |
-| `float32/64` | Supported |
-| `decimal128` | Precision 1 through 38, with Scale from 0 through Precision |
-| `utf8`, `large_utf8` | Supported |
-| `date32(day)` | Supported |
-| Timezone-naive `timestamp(s/ms/us)` | Supported |
-
-Predicates on other readable types, including `float16`, `decimal256`, Binary,
`date64`, Time, nanosecond Timestamp, timezone-aware Timestamp, and complex
types, currently remain in Doris.
-
-### Operators Supported for Pushdown
-
-| SQL Predicate | Pushdown Condition |
-|---|---|
-| `=`, `!=`, `<>`, `<`, `<=`, `>`, `>=` | Direct comparison between a column
and a constant. The constant may be on the left side. |
-| `<=>` | Direct null-safe equality comparison between a column and a
constant; preserves a non-`NULL`, two-valued result inside `NOT`, `AND`, or
`OR` |
-| `IN`, `NOT IN` | Non-empty constant list that does not contain `NULL` |
-| `IS NULL`, `IS NOT NULL` | Direct column reference |
-| `AND` | Top-level conjuncts can be pushed down independently, with
unsupported conjuncts retained in Doris |
-| `OR` | Both branches must be fully convertible |
-| `NOT` | The operand must be fully convertible |
-
-The following forms are generally not pushed down:
-
-- Functions or arithmetic expressions applied to a column.
-- An empty `IN` list or an `IN` list containing `NULL`.
-- An `OR` or `NOT` expression in which only part of the expression can be
converted.
-- A data type or constant value that cannot be converted to Lance without loss.
-
-Use `lancePushdownPredicate` in `EXPLAIN` to inspect the conditions that are
actually pushed down:
-
-```sql
-EXPLAIN
-SELECT user_id
-FROM lance_catalog.default.user_profiles
-WHERE age >= 18 AND country IN ('CN', 'US');
-```
-
-## Query Lance with File TVFs
-
-If you only need to read a Lance dataset at a known path, you can use the
`s3()` or `local()` TVF without creating a Catalog. `uri` or `file_path` must
point to the root directory of a Lance dataset, rather than an internal data
file.
-
-### S3 TVF
-
-```sql
-SELECT user_id, name
-FROM s3(
- "uri" = "s3://my-bucket/lance/user_profiles.lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true",
- "format" = "lance"
-)
-WHERE user_id > 100;
-```
-
-For an S3 TVF, the FE obtains the Schema, current version, and Fragment list.
Doris pins that version and scans its Fragments in parallel.
-
-### Local TVF
-
-```sql
-SELECT user_id, name
-FROM local(
- "file_path" = "/data/lance/user_profiles.lance",
- "backend_id" = "10001",
- "format" = "lance"
-);
-```
-
-`file_path` is passed as written to the Lance Reader on the target BE. Doris
does not prepend `user_files_secure_path` or expand this path as a Glob, so it
must point directly to the root directory of one Lance dataset that the target
BE can access. An absolute path is recommended.
-
-Local TVF Schema discovery and execution each open the latest dataset version
independently. The version resolved during Schema discovery is not currently
pinned for the subsequent scan. If the dataset changes between query analysis
and execution, the discovered Schema and scanned snapshot can differ. Avoid
modifying the dataset while a Local TVF query is being analyzed and executed.
The current Local Lance TVF uses one Scanner.
-
-Lance file TVFs have the following additional limitations:
-
-- Only `s3()` and `local()` are supported. Other file TVFs, such as HDFS and
HTTP, are not currently supported.
-- `path_partition_keys` is not supported.
-- One TVF path can represent only one Lance dataset.
-- `DESC FUNCTION` can display a Schema containing unsupported types, but SQL
cannot project unsupported columns.
-
-## Vector Search
-
-`vector_search()` is a relational TVF that performs Top-K search on a vector
column in a Lance table. It can use an existing Lance vector index or perform
Flat Search.
-
-### Syntax and Example
-
-```sql
-SELECT user_id, label, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10",
- "offset" = "3",
- "metric" = "l2",
- "nprobes" = "20",
- "refine_factor" = "10",
- "filter" = "category = 'book'",
- "use_index" = "true"
-)
-ORDER BY _distance ASC, user_id;
-```
-
-The relation schema of `vector_search()` contains all columns from the Lance
source table plus the `_distance` column generated by the Lance Scanner for the
nearest-neighbor query. The final SQL result contains only columns projected by
`SELECT`. Doris exposes `_distance` as `FLOAT`. It is a distance, not a generic
similarity score: a lower value means that two vectors are closer. The source
table must not already contain a column named `_distance`. A SQL relation does
not guarantee fina [...]
-
-`table` must parse as exactly three `catalog.database.table` name parts. A
multilevel Lance Namespace maps to one Doris database name containing `.`, so
quote the database part with backticks. For example, use the following value
for table `items` in Namespace `doris.analytics`:
-
-```sql
-"table" = "lance_catalog.`doris.analytics`.items"
-```
-
-Do not use the unquoted form `lance_catalog.doris.analytics.items`; it parses
as four name parts and is rejected. A table-only name or `database.table` name
is also rejected.
-
-### Parameters
-
-| Parameter | Required | Default | Description |
-|---|---|---|---|
-| `table` | Yes | - | Fully qualified, three-part `catalog.database.table`
name. If a multilevel Namespace maps to a database name containing `.`, quote
the database part with backticks. It must identify a table in a Lance Catalog,
and the user must have the `SELECT` privilege on the table. |
-| `column` | Yes | - | Vector column name.
`fixed_size_list<float16\|float32\|float64\|uint8\|int8>` is currently
supported. |
-| `query_vector` | Yes | - | JSON number array. Its dimension must match the
vector column, and each value must be representable by the vector element type.
|
-| `top_k` | No | `10` | Number of results returned after skipping `offset`. It
must be a positive integer. |
-| `offset` | No | `0` | Number of nearest neighbors skipped inside the vector
search. It must be a non-negative integer. `top_k + offset` must not exceed the
maximum unsigned 32-bit integer. |
-| `metric` | No | Metric of the matching index; without an index, `hamming`
for `uint8` and `l2` for other supported types | Distance metric: `l2`,
`cosine`, `dot`, or `hamming`. `dot_product` is an alias for `dot`. `uint8`
vectors support only `hamming`; the other currently supported vector element
types support `l2`, `cosine`, and `dot`. |
-| `filter` | No | - | Lance SQL condition evaluated before vector candidates
are generated; that is, a Prefilter. |
-| `nprobes` | No | Minimum `1`, with no maximum | Number of IVF index
partitions to probe. It must be a positive integer. When unset, Lance starts
with one partition and can probe additional partitions when a Prefilter leaves
too few candidates. Setting it explicitly to `N` fixes both the minimum and
maximum number of probes to `N`. |
-| `refine_factor` | No | Refinement disabled | Candidate refinement
multiplier. It must be a positive integer. When unset, Lance does not recompute
distances from the original vectors, so `_distance` from a quantized index may
be approximate. When set to `N`, Lance first retrieves `(top_k + offset) × N`
candidates, recomputes their exact distances from the original vectors, and
reorders them. Setting it to `1` still enables refinement and therefore differs
from leaving it unset. |
-| `ef` | No | `floor(1.5 × (top_k + offset))` | Candidate width retained
during HNSW graph search. It must be a positive integer. If `refine_factor` is
also set, the default is `floor(1.5 × (top_k + offset) × refine_factor)`. It
has no effect on non-HNSW indexes. |
-| `use_index` | No | `true` | When `true`, Doris plans compatible physical
Lance index segments as indexed splits and keeps uncovered Fragments as Flat
Search splits. If no usable compatible index metadata is available, Doris falls
back to Fragment splits. When `false`, Doris creates one split per visible
Fragment and forces Flat Search. |
-
-These defaults correspond to the Lance Scanner behavior currently integrated
with Doris. When `metric` is omitted, Doris uses the metric configured when a
compatible vector index was created. If there is no compatible index, or if
`"use_index" = "false"`, `uint8` vectors use `hamming`, while the other
currently supported vector element types use `l2`.
-
-### Supported Vector Index Types
-
-The embedded `lance-c v0.1.6` explicitly supports the following Lance vector
index combinations:
-
-| Index type | Description | Main query parameters |
-|---|---|---|
-| `IVF_FLAT` | IVF partitions with original-vector distance computation inside
each partition | `nprobes` |
-| `IVF_SQ` | IVF with Scalar Quantization | `nprobes`, `refine_factor` |
-| `IVF_PQ` | IVF with Product Quantization | `nprobes`, `refine_factor` |
-| `IVF_HNSW_FLAT` | IVF with HNSW whose graph nodes retain original vectors |
`nprobes`, `ef` |
-| `IVF_HNSW_SQ` | IVF and HNSW with Scalar Quantization | `nprobes`, `ef`,
`refine_factor` |
-| `IVF_HNSW_PQ` | IVF and HNSW with Product Quantization | `nprobes`, `ef`,
`refine_factor` |
-
-`vector_search()` only queries indexes. It does not create an index in Doris
and does not expose an index-type or index-name parameter. With
`use_index=true`, the FE reads vector-index metadata from the pinned dataset
snapshot and selects a logical index compatible with the vector column and
metric. It then assigns each physical segment of that logical index that still
covers visible data to an indexed Scan Split. Each indexed Split carries the
segment UUID and the currently visible Frag [...]
-
-A logical Lance index can contain multiple physical index segments, and one
physical segment can cover multiple Fragments. Fragments not covered by the
selected index are not omitted: Doris adds one fallback Split for each such
Fragment, which uses Flat Search. If the FE cannot construct a usable
index-segment plan, it falls back to Fragment-level splits. With
`use_index=false`, Doris skips index metadata planning and forces Flat Search
for every visible Fragment. Flat Search is not an A [...]
-
-### Prefilter and Post-Filter
-
-The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the
Lance Scanner for each search Split, and Lance evaluates it before ANN or Flat
Search generates candidates:
-
-```sql
-SELECT user_id, category, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10",
- "filter" = "category = 'book'"
-)
-ORDER BY _distance ASC, user_id;
-```
-
-Lance reads and evaluates columns referenced only by `filter` internally. If
such a column is not referenced by `SELECT` or another Doris expression, it
does not have to be returned to Doris.
-
-An outer `WHERE` is a **Post-filter**. The optimizer moves it into the Doris
Lance Scan, but does not convert it into a Lance Prefilter. It runs after Lance
generates candidates for each search Split and before Doris performs its local
and global TopN operations.
-
-```sql
-SELECT user_id, category, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10"
-)
-WHERE category = 'book'
-ORDER BY _distance ASC, user_id;
-```
-
-Consequently, an outer `WHERE` only filters candidates that have already been
generated and does not cause Lance to replenish them. The final result may
contain fewer than `top_k` rows. If the filter must reduce the vector search
space and nearest neighbors must be selected from the filtered rows, use the
TVF `filter` parameter.
-
-### Current Execution Model
-
-`vector_search()` uses distributed candidate search instead of one Scanner for
the entire dataset. Its Split boundary depends on index coverage:
-
-1. During planning, the FE pins a positive Lance dataset snapshot version and
reads the visible Fragments in that snapshot. When `use_index=true`, it also
reads vector-index metadata.
-2. If a compatible logical vector index has usable segment coverage, each
physical index segment that still covers visible Fragments becomes one indexed
Scan Split. The Split contains that segment's UUID and the intersection of its
Fragment bitmap with the visible Fragments in the pinned snapshot. A Split can
therefore contain multiple Fragment IDs.
-3. Every visible Fragment not covered by those indexed Splits becomes an
independent fallback Fragment Split. This keeps data appended after index
creation searchable without requiring the index to be optimized first. If no
usable index-segment plan exists, all visible Fragments use Fragment splits.
With `use_index=false`, all visible Fragments use Flat Search splits directly.
-4. For query parameters `top_k=K` and `offset=n`, every indexed or fallback
Split requests at most `K+n` candidates and does not apply the offset locally.
An indexed Split searches only its assigned physical index segment; a fallback
Fragment Split performs Flat Search for its Fragment. Lance evaluates the TVF
`filter` before generating candidates, while Doris Scan evaluates an outer
`WHERE` afterward.
-5. Doris performs local TopN, Exchange, and global TopN over candidates from
all Splits, merging by `_distance ASC`. Only the global TopN applies
`offset=n`: it skips the first `n` rows and returns `K` rows.
-
-A Split-level candidate set therefore only supplies candidates for global
merging and is not the final result. Index-segment splits, fallback Fragment
splits, and any later Row-ID fetches all use the same pinned snapshot
throughout the query. Refreshing index coverage changes how newly appended
Fragments are searched, but uncovered Fragments remain part of the result space
through Flat Search.
-
-The execution order is:
-
-```text
-Pinned dataset snapshot
- -> FE Split planning
- -> Indexed coverage: one Split per physical Index Segment -> ANN Search
- -> Uncovered or unindexed data: one Split per Fragment -> Flat Search
- -> Per Split: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates
- -> Doris Scan Post-filter
- -> Doris local TopN
- -> Exchange
- -> Doris global TopN (applies offset=n and limit=K)
- -> Optional lazy-materialization Fetch
-```
-
-### Two-Phase TopN Read and Lazy Materialization
-
-`vector_search()` can use a two-phase read when
`experimental_topn_lazy_materialization_threshold` is greater than `0`, `top_k`
does not exceed the threshold, and at least one top-level column is eligible
for deferred reading. The default threshold is `1024`. Phase 1 carries only the
columns required for candidate filtering and TopN, plus an internal Row
Location. After global TopN, Phase 2 reads the other output columns only for
the retained rows.
-
-For example, assume the source table has these columns:
-
-| Column | Purpose |
-|---|---|
-| `user_id` | Final output column |
-| `category` | Post-filter column used by the outer `WHERE` |
-| `title`, `payload` | Final output columns |
-| `embedding` | Lance vector-search column |
-
-Run this query with `K=10` and `n=3`:
-
-```sql
-SET experimental_topn_lazy_materialization_threshold = 1024;
-
-SELECT user_id, title, payload, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10",
- "offset" = "3"
-)
-WHERE category = 'book';
-```
-
-A typical two-phase column flow is:
-
-| Stage or operator | Columns read or output | Description |
-|---|---|---|
-| Lance Split Search | Uses `embedding` internally; returns `_distance`,
`category`, and the internal Lance Row ID to Doris | `embedding` participates
in ANN/Flat Search but is not returned as a result column unless SQL projects
it. Each Index Segment or fallback Fragment Split produces at most `K+n`
candidates. |
-| Doris Scan Post-filter | `_distance`, `category`, and the internal Row
Location | Evaluates `category = 'book'`. A column used by an outer `WHERE`
must remain in Phase 1. Doris encodes the Lance Row ID and dataset mapping into
an internal Row Location, which Fetch resolves to the same pinned snapshot. |
-| Local and global TopN | Phase-1 required columns and the internal Row
Location | Global TopN merges by `_distance` and applies `offset=n` and
`limit=K`. |
-| Row ID Fetch | Uses the internal Row Location to read `user_id`, `title`,
and `payload` | Reads deferred columns for rows retained by global TopN from
the same Lance dataset snapshot without rescanning Fragments. |
-| Final Materialize | `user_id`, `title`, `payload`, and `_distance` |
Combines deferred columns with columns retained from Phase 1 to produce the
final SQL output. |
-
-Phase-1 required columns are not limited to `_distance` and Post-filter
columns. Any column referenced by a Doris expression or operator before global
TopN is operative and must be read in Phase 1. For example, adding `ORDER BY
_distance, user_id` makes `user_id` a Phase-1 column, so it cannot be deferred
to the Row-ID Fetch. Nested subcolumn projections are also not currently
deferred. A top-level column used only by the final projection can be fetched
in Phase 2.
-
-A Prefilter column referenced in the TVF `filter` differs from an outer
`WHERE` column. Lance uses the former internally during search, so appearing in
the `filter` string alone does not require the column to be returned to Doris.
Doris Scan evaluates the latter, so it must be present in Phase 1.
-
-Setting `experimental_topn_lazy_materialization_threshold` to `-1` disables
the two-phase read. A single-phase read is also used when `top_k` exceeds the
threshold or no column can be deferred. Single-phase mode returns all
query-required output columns from Scan, but vector search still generates
candidates in parallel per Index Segment or fallback Fragment Split and Doris
still merges a global TopN. It does not become an ordinary Doris full-column
table scan. Indexed splits use their a [...]
-
-## Current Limitations and Recommendations
-
-- Lance Catalogs and Lance TVFs are read-only. `CREATE TABLE`, `INSERT`,
`UPDATE`, `DELETE`, `TRUNCATE TABLE`, and writing data back to Lance are not
supported.
-- Queries always read the current version selected during planning. SQL cannot
select a Version or perform Time Travel.
-- For tables containing unsupported column types, explicitly list the columns
to read instead of projecting unsupported columns through `SELECT *`.
-- For regular scans, inspect `lancePushdownPredicate` in `EXPLAIN` to verify
which conditions have been pushed down.
-- Create a vector index in Lance that matches the intended query before
running indexed vector search. For small datasets or validation, set
`"use_index" = "false"` to perform Flat Search.
-- For deterministic vector result ordering, explicitly use `ORDER BY _distance
ASC` and add a unique tie-breaker.
-- Use the `vector_search()` `filter` parameter when filtering must occur
before vector candidates are generated. An outer `WHERE` filters only the
candidates already generated by each search Split and runs before Doris global
TopN, so allow for a final result with fewer than `top_k` rows.
-- Use `EXPLAIN` to inspect `lanceSearchFragments` and
`lanceSearchIndexSegments`. The former is the number of visible Fragments in
the pinned snapshot; the latter is the number of physical Index Segment splits
selected by the FE. Additional fallback Fragment splits may also be present.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx
deleted file mode 100644
index 972212034ce..00000000000
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/lance-catalog.mdx
+++ /dev/null
@@ -1,541 +0,0 @@
----
-{
- "title": "Lance Catalog",
- "language": "zh-CN",
- "description": "Apache Doris Lance Catalog 使用指南：通过 Filesystem 或 REST
Namespace 访问 Lance 数据集，支持并行读取、谓词下推、S3/Local TVF 以及向量检索。"
-}
----
-
-:::note
-这是一个实验性功能。
-
-Lance Catalog 自 Apache Doris 4.2 版本开始支持。
-:::
-
-Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lance Catalog 发现 Lance Namespace
中的数据库和表，并直接查询存储在本地文件系统或 S3 兼容对象存储中的 Lance 数据集。
-
-当前 Doris 对 Lance 提供只读能力，不支持创建、写入、更新或删除 Lance 表。
-
-## 功能概览
-
-| 功能 | 支持情况 |
-|---|---|
-| Filesystem Catalog | 支持本地文件系统、`file://` 和 `s3://` Warehouse |
-| REST Catalog | 支持 Lance REST Namespace，以及无认证、Bearer Token、API Key 和自定义 HTTP
Header |
-| 元数据访问 | 支持 `SHOW DATABASES`、`SHOW TABLES` 和 `DESC` |
-| 数据查询 | 支持列裁剪、并行扫描 Lance Fragment 和当前版本的快照一致性读取 |
-| 谓词下推 | 支持将部分标量谓词下推到 Lance 执行 |
-| 文件 TVF | 支持通过 `s3()` 和 `local()` 直接查询 Lance 数据集 |
-| 向量检索 | 使用物理 Lance Index Segment 作为并行 Split，对未覆盖的 Fragment 保留 Flat Search
Split，并由 Doris 合并全局 Top-K |
-| 写入 Lance | 暂不支持 |
-| Time Travel | 暂不支持 |
-| Full-Text Search / Hybrid Search | 暂不支持 |
-
-## Lance 版本与兼容性
-
-Doris BE 数据读取器使用 `lance-c v0.1.6` 构建。该版本在 Doris 中绑定的 Lance 源码版本为
`9.1.0-beta.3`（Lance commit `e934cc2c`）。`lance-c` 和 Lance Rust crates 的版本表示
Doris 集成的读取器实现版本，与数据集中记录的 Lance `data_storage_version` 不是同一个概念。
-
-当前读取器的文件格式兼容情况如下：
-
-| `data_storage_version` | 读取支持 | 说明 |
-|---|---|---|
-| `0.1` / `legacy` | 支持 | Lance 的初始文件格式。 |
-| `2.0`（写入选项别名 `0.3`） | 支持 | Lance v2 文件格式的早期版本。 |
-| `2.1` / `stable` | 支持，默认稳定格式 | 当前内置 Lance 的 `stable` 写入选项和新数据集默认格式均解析为
`2.1`。 |
-| `2.2` | 支持 | 当前内置 Lance 将其视为稳定格式，但它不是默认写入格式。 |
-| `2.3` / `next` | 实验性，不保证兼容 | 当前内置 Lance 将 `2.3` 标记为不稳定格式，`next` 写入选项解析为
`2.3`。 |
-| 后续或未知版本 | 不支持 | 打开或扫描数据集时可能返回不支持存储版本的错误。 |
-
-Lance SDK 发行版本号和文件格式版本相互独立。无论数据集由更早还是更新的 Lance SDK 写入，只有当其存储格式、必需的表级 Feature
Flag、索引格式以及 Arrow/Lance 数据类型均可被 Doris 内置版本识别时，Doris 才能读取。因此：
-
-- Doris 预期能够读取使用 `0.1`、`2.0`、`2.1` 和 `2.2` 存储格式写入的数据集，同时还需满足下文所述的数据类型限制。
-- 不保证向前兼容。更新的 Lance 版本写入或修改数据集后，如果使用了更新的存储格式、未知的必需 Manifest Feature、新索引格式或不支持的
Extension 类型，Doris 可能无法读取。
-- 对于必须由当前 Doris 版本持续读取的数据集，建议使用当前默认稳定格式 `2.1`，且不要使用 `next`。引入更新的写入器或可选 Lance
功能后，应先使用目标 Doris 版本验证生成的数据集，再用于生产环境。
-
-## 配置 Catalog
-
-### 语法
-
-```sql
-CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "<filesystem|rest>",
- {CatalogProperties},
- {StorageProperties},
- {CommonProperties}
-);
-```
-
-### 通用属性
-
-| 属性 | 是否必需 | 默认值 | 说明 |
-|---|---|---|---|
-| `type` | 是 | - | 固定为 `lance`。 |
-| `lance.catalog.type` | 否 | `filesystem` | Catalog 类型，可选值为 `filesystem` 或
`rest`。 |
-| `lance.namespace.parent` | 否 | 空 | 仅访问指定 Lance Namespace 及其子
Namespace。例如默认分隔符下，`production$analytics` 表示两级 Namespace。 |
-| `lance.namespace.delimiter` | 否 | `$` | 用于解析 `lance.namespace.parent`，同时会传递给
REST Namespace 客户端。该配置不改变 Doris 中多级 Namespace 的展示方式。 |
-| `lance.namespace.root_database` | 否 | `default` | Lance 根 Namespace 在 Doris
中映射的数据库名。 |
-
-### Filesystem Catalog
-
-Filesystem Catalog 直接从 Warehouse 目录发现 Lance Namespace 和表。
-
-| 属性 | 是否必需 | 说明 |
-|---|---|---|
-| `warehouse` | 是 | Lance Warehouse 根路径。支持本地绝对路径、`file://` URI 和 `s3://` URI。 |
-
-#### 使用 S3 兼容对象存储
-
-下面以 MinIO 为例创建 Catalog：
-
-```sql
-CREATE CATALOG lance_catalog PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "s3://my-bucket/lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true"
-);
-```
-
-访问 AWS S3 时，可以省略 `s3.endpoint`，并按实际环境配置访问密钥、Region 和 Path Style。
-
-#### 使用本地文件系统
-
-```sql
-CREATE CATALOG lance_local PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "/data/lance"
-);
-```
-
-使用本地文件系统时，`warehouse` 必须是绝对路径。FE 需要通过该路径读取 Namespace 和表元数据，执行查询的 BE
也需要能够通过相同路径访问数据。因此在多节点环境中，应将相同的共享目录挂载到所有相关 FE 和 BE 节点。
-
-### REST Catalog
-
-REST Catalog 通过 Lance REST Namespace 获取 Namespace、表地址和存储访问参数。REST Catalog
不需要、也不允许设置 `warehouse`。
-
-| 属性 | 是否必需 | 默认值 | 说明 |
-|---|---|---|---|
-| `lance.rest.uri` | 是 | - | REST 服务地址，必须使用 `http://` 或 `https://`。 |
-| `lance.rest.security.type` | 否 | `none` | 认证方式，可选值为 `none`、`bearer` 或
`api_key`。 |
-| `lance.rest.bearer-token` | 使用 Bearer 认证时是 | - | Bearer Token。 |
-| `lance.rest.api-key` | 使用 API Key 认证时是 | - | API Key，通过 `x-api-key` Header
发送。 |
-| `lance.rest.header.<header-name>` | 否 | - | 发送给 REST 服务的自定义 HTTP Header。认证
Header 应使用上面的专用认证属性配置。 |
-
-使用 Bearer Token 创建 REST Catalog：
-
-```sql
-CREATE CATALOG lance_rest PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "rest",
- "lance.rest.uri" = "https://lance.example.com";,
- "lance.rest.security.type" = "bearer",
- "lance.rest.bearer-token" = "your-token"
-);
-```
-
-使用 API Key 时，将认证配置替换为：
-
-```sql
-"lance.rest.security.type" = "api_key",
-"lance.rest.api-key" = "your-api-key"
-```
-
-如果 REST 服务返回临时存储凭证，Doris 会使用这些凭证访问对应的 Lance 表。也可以在 Catalog 中配置
`s3.endpoint`、`s3.access_key`、`s3.secret_key`、`s3.region` 和
`use_path_style`，作为默认的对象存储访问参数。
-
-:::caution
-当前 BE Reader 不支持由 REST Namespace 管理版本的 Lance 表（Managed Versioning）。
-:::
-
-## Namespace 映射
-
-Lance 支持多级 Namespace，而 Doris Catalog 使用数据库名承载 Namespace：
-
-| Lance Namespace | Doris 数据库名 |
-|---|---|
-| 根 Namespace | `default`，可通过 `lance.namespace.root_database` 修改 |
-| `doris` | `doris` |
-| `doris.analytics` | `doris.analytics` |
-
-多级 Namespace 在 Doris 中使用 `.` 连接为一个数据库名。引用包含 `.` 的数据库名时，需要使用反引号：
-
-```sql
-SHOW TABLES FROM lance_catalog.`doris.analytics`;
-
-SELECT *
-FROM lance_catalog.`doris.analytics`.user_features;
-```
-
-`lance.namespace.parent` 可用于限定 Catalog 可见的 Namespace 子树。例如：
-
-```sql
-CREATE CATALOG lance_analytics PROPERTIES (
- "type" = "lance",
- "lance.catalog.type" = "filesystem",
- "warehouse" = "s3://my-bucket/lance",
- "lance.namespace.parent" = "production$analytics",
- "s3.region" = "us-east-1"
-);
-```
-
-此时 Doris 只展示 `production.analytics` 之下的表和子 Namespace。
-
-## 查询 Lance 表
-
-创建 Catalog 后，可以像查询普通外表一样浏览并查询 Lance 表：
-
-```sql
-SHOW DATABASES FROM lance_catalog;
-
-SHOW TABLES FROM lance_catalog.default;
-
-DESC lance_catalog.default.user_profiles;
-
-SELECT user_id, name, age
-FROM lance_catalog.default.user_profiles
-WHERE age >= 18
-ORDER BY user_id
-LIMIT 100;
-```
-
-也可以将 Lance 数据写入 Doris 内表：
-
-```sql
-INSERT INTO internal.demo.user_profiles
-SELECT user_id, name, age
-FROM lance_catalog.default.user_profiles;
-```
-
-普通 Catalog 查询会在规划阶段固定一个 Lance 数据集版本，并按 Fragment 生成扫描任务。因此，同一条查询读取一致的快照，同时可以由多个
Scanner 并行扫描不同 Fragment，不会让每个 Scanner 重复扫描整个数据集。
-
-## 类型映射
-
-| Lance / Arrow 类型 | Doris 类型 | 说明 |
-|---|---|---|
-| `bool` | `BOOLEAN` | |
-| `int8` | `TINYINT` | |
-| `uint8` | `SMALLINT` | 无符号整数无损提升 |
-| `int16` | `SMALLINT` | |
-| `uint16` | `INT` | 无符号整数无损提升 |
-| `int32` | `INT` | |
-| `uint32` | `BIGINT` | 无符号整数无损提升 |
-| `int64` | `BIGINT` | |
-| `uint64` | `LARGEINT` | 无符号整数无损提升 |
-| `float16` | `FLOAT` | 提升为 32 位浮点数 |
-| `float32` | `FLOAT` | |
-| `float64` | `DOUBLE` | |
-| `decimal128(P,S)` | `DECIMAL(P,S)` | 最大精度为 38 |
-| `decimal256(P,S)` | `DECIMAL(P,S)` | 最大精度为 76 |
-| `utf8`、`large_utf8` | `TEXT` | |
-| `binary`、`large_binary` | `VARBINARY(2147483647)` | |
-| `fixed_size_binary(N)` | `VARBINARY(N)` | 保留固定字节宽度 |
-| `date32(day)`、`date64(ms)` | `DATE` | `date64` 应表示完整自然日 |
-| `time32(s)` | `TIME(0)` | |
-| `time32(ms)` | `TIME(3)` | |
-| `time64(us)`、`time64(ns)` | `TIME(6)` | 纳秒精度截断为微秒 |
-| 无时区 `timestamp(s)` | `DATETIME` | 不随 Session Time Zone 转换 |
-| 无时区 `timestamp(ms)` | `DATETIME(3)` | 不随 Session Time Zone 转换 |
-| 无时区 `timestamp(us)`、`timestamp(ns)` | `DATETIME(6)` | 纳秒精度截断为微秒 |
-| 带时区 `timestamp` | `TIMESTAMPTZ(0-6)` | 保存时间点，按 Doris Session Time Zone 展示 |
-| `struct` | `STRUCT` | 子字段递归映射 |
-| `list`、`large_list`、`fixed_size_list` | `ARRAY` | 元素类型递归映射 |
-| `map` | `MAP` | Key 和 Value 类型递归映射 |
-
-当前不支持以下类型：
-
-- Arrow `null` 和 `duration`。
-- 带有 `ARROW:extension:name` 元数据的 Arrow/Lance Extension 类型，例如 Lance Blob
v2、Arrow JSON Extension 和 Lance BFloat16 Extension。
-- 无法递归映射其子类型的复杂类型。
-- 保留了 Dictionary 标记的 Arrow Dictionary 类型。
-
-对于不支持的顶层列，Catalog 表的 `DESC` 和 Lance 文件 TVF 的 `DESC FUNCTION` 都会保留该列，并显示
`unknown type:
UNSUPPORTED_TYPE`。如果复杂类型的任一子字段无法映射，则整个顶层复杂列会标记为不支持。查询只投影支持的列仍可正常执行；当 SQL
投影不支持的列时，Doris 会在分析阶段报错。例如：
-
-```sql
-SELECT * EXCEPT(blob_col, json_col)
-FROM lance_catalog.default.all_types;
-```
-
-:::note
-部分 Lance Java SDK 版本可能在读取 Schema 时丢失 Dictionary 标记，并将 Dictionary
列暴露为物理索引类型。该结果不代表 Doris 已支持 Dictionary 的逻辑值，请勿依赖这一行为。
-:::
-
-## 谓词下推
-
-Doris 会将语义兼容的谓词转换为 Substrait 表达式，并交给 Lance 在读取阶段执行。对于已完整下推的条件，Doris BE
不再重复计算该条件；不能安全下推的条件仍由 Doris 执行。
-
-### 支持下推的数据类型
-
-| Lance / Arrow 类型 | 下推范围 |
-|---|---|
-| `bool` | 等值、空值和逻辑运算，不包含大小比较 |
-| `int8/16/32/64` | 支持 |
-| `uint8/16/32/64` | 支持 |
-| `float32/64` | 支持 |
-| `decimal128` | 精度 1-38，Scale 范围为 0 到 Precision |
-| `utf8`、`large_utf8` | 支持 |
-| `date32(day)` | 支持 |
-| 无时区 `timestamp(s/ms/us)` | 支持 |
-
-其他可读取类型，例如 `float16`、`decimal256`、Binary、`date64`、Time、纳秒 Timestamp、带时区
Timestamp 和复杂类型，当前保留为 Doris 侧谓词。
-
-### 支持下推的操作符
-
-| SQL 谓词 | 下推条件 |
-|---|---|
-| `=`、`!=`、`<>`、`<`、`<=`、`>`、`>=` | 直接的列与常量比较；支持将常量写在左侧 |
-| `<=>` | 直接的列与常量 Null-safe 等值比较；在 `NOT`、`AND` 或 `OR` 中仍保持非 `NULL` 的二值逻辑结果 |
-| `IN`、`NOT IN` | 非空常量列表，列表中不能包含 `NULL` |
-| `IS NULL`、`IS NOT NULL` | 直接引用列 |
-| `AND` | 顶层 Conjunct 可以分别下推，不能下推的部分保留在 Doris |
-| `OR` | 两个分支都能完整转换时下推 |
-| `NOT` | 操作数能完整转换时下推 |
-
-以下形式通常不会下推：
-
-- 列上包含函数或算术表达式。
-- `IN` 列表为空或包含 `NULL`。
-- `OR` 或 `NOT` 中只有部分表达式可转换。
-- 数据类型或常量值无法无损转换到 Lance。
-
-可以通过 `EXPLAIN` 中的 `lancePushdownPredicate` 查看实际下推的条件：
-
-```sql
-EXPLAIN
-SELECT user_id
-FROM lance_catalog.default.user_profiles
-WHERE age >= 18 AND country IN ('CN', 'US');
-```
-
-## 使用文件 TVF 查询 Lance
-
-如果只需要读取一个已知路径的 Lance 数据集，可以不创建 Catalog，直接使用 `s3()` 或 `local()` TVF。`uri` 或
`file_path` 必须指向 Lance 数据集的根目录，而不是其内部的数据文件。
-
-### S3 TVF
-
-```sql
-SELECT user_id, name
-FROM s3(
- "uri" = "s3://my-bucket/lance/user_profiles.lance",
- "s3.endpoint" = "http://127.0.0.1:9000";,
- "s3.access_key" = "admin",
- "s3.secret_key" = "password",
- "s3.region" = "us-east-1",
- "use_path_style" = "true",
- "format" = "lance"
-)
-WHERE user_id > 100;
-```
-
-S3 TVF 在 FE 获取 Schema、当前版本和 Fragment 列表，并固定该版本后按 Fragment 并行扫描。
-
-### Local TVF
-
-```sql
-SELECT user_id, name
-FROM local(
- "file_path" = "/data/lance/user_profiles.lance",
- "backend_id" = "10001",
- "format" = "lance"
-);
-```
-
-`file_path` 会按用户填写的值直接传给目标 BE 上的 Lance Reader。Doris 不会自动拼接
`user_files_secure_path`，也不会对该路径执行 Glob 展开，因此该参数必须直接指向目标 BE 可访问的单个 Lance
数据集根目录；建议使用绝对路径。
-
-Local TVF 的 Schema 发现和执行扫描会分别打开数据集的最新版本，当前不会将 Schema
发现时解析出的版本固定到后续扫描。如果数据集在查询分析与执行之间发生更新，Schema 和实际扫描的快照可能不一致。因此，应避免在 Local TVF
查询的分析和执行期间修改数据集。当前 Local Lance TVF 使用单个 Scanner。
-
-Lance 文件 TVF 还有以下限制：
-
-- 仅支持 `s3()` 和 `local()`，暂不支持 HDFS、HTTP 等其他文件 TVF。
-- 不支持 `path_partition_keys`。
-- 一个 TVF 路径只能表示一个 Lance 数据集。
-- `DESC FUNCTION` 可以展示包含不支持类型的 Schema，但 SQL 不能投影不支持的列。
-
-## 向量检索
-
-`vector_search()` 是一个关系型 TVF，用于对 Lance 表的向量列执行 Top-K 检索。它既可以使用 Lance
中已建立的向量索引，也可以执行 Flat Search。
-
-### 语法和示例
-
-```sql
-SELECT user_id, label, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10",
- "offset" = "3",
- "metric" = "l2",
- "nprobes" = "20",
- "refine_factor" = "10",
- "filter" = "category = 'book'",
- "use_index" = "true"
-)
-ORDER BY _distance ASC, user_id;
-```
-
-`vector_search()` 的关系 Schema 包含 Lance 源表的所有列，以及 Lance Scanner 为最近邻查询生成的
`_distance` 列；最终 SQL 结果只包含 `SELECT` 投影的列。Doris 将 `_distance` 作为 `FLOAT`
提供给用户。它表示距离而不是通用的相似度分数，值越小表示两个向量越接近。源表不能已经包含名为 `_distance` 的列。SQL
关系本身不保证最终展示顺序，因此需要稳定的最近邻顺序时，应显式使用 `ORDER BY _distance ASC`，并建议增加唯一列作为距离相同情况下的
Tie-breaker。
-
-`table` 必须解析为恰好三部分的 `catalog.database.table` 名称。多级 Lance Namespace 在 Doris
中映射为包含 `.` 的单个数据库名，因此必须使用反引号将数据库部分括起来。例如，表 `items` 位于 `doris.analytics`
Namespace 时，应写为：
-
-```sql
-"table" = "lance_catalog.`doris.analytics`.items"
-```
-
-不要写成未引用的 `lance_catalog.doris.analytics.items`，因为它会被解析为四部分名称并报错。只填写表名或
`database.table` 也会报错。
-
-### 参数
-
-| 参数 | 是否必需 | 默认值 | 说明 |
-|---|---|---|---|
-| `table` | 是 | - | 完整的三部分 `catalog.database.table` 名称。多级 Namespace 对应的数据库名包含
`.` 时，必须使用反引号引用数据库部分。该表必须属于 Lance Catalog，用户需要拥有该表的 `SELECT` 权限。 |
-| `column` | 是 | - | 向量列名。当前支持
`fixed_size_list<float16\|float32\|float64\|uint8\|int8>`。 |
-| `query_vector` | 是 | - | JSON 数字数组。维度必须与向量列一致，元素值必须能由向量元素类型表示。 |
-| `top_k` | 否 | `10` | 跳过 `offset` 后返回的结果数，必须为正整数。 |
-| `offset` | 否 | `0` | 在向量检索内部跳过的最近邻数量，必须为非负整数。`top_k + offset` 不能超过无符号 32
位整数上限。 |
-| `metric` | 否 | 匹配索引的 Metric；无索引时 `uint8` 为 `hamming`，其他支持类型为 `l2` |
距离类型：`l2`、`cosine`、`dot` 或 `hamming`。`dot_product` 是 `dot` 的别名。`uint8` 向量仅支持
`hamming`；其他当前支持的向量元素类型支持 `l2`、`cosine` 和 `dot`。 |
-| `filter` | 否 | - | Lance SQL 条件，在生成候选向量之前执行，即 Prefilter。 |
-| `nprobes` | 否 | 最少 `1`，不限制最大值 | IVF 索引探测的分区数量，必须为正整数。不设置时从 1 个分区开始；使用
Prefilter 且候选不足时，Lance 可以继续探测更多分区。显式设置为 `N` 时，最少和最多探测数都会固定为 `N`。 |
-| `refine_factor` | 否 | 不启用精排 | 候选集精排倍数，必须为正整数。不设置时不基于原始向量重新计算距离，量化索引返回的
`_distance` 可能是近似距离；设置为 `N` 后，Lance 先获取 `(top_k + offset) × N`
个候选，再用原始向量计算真实距离并重新排序。即使设置为 `1` 也会执行精排，因此与不设置不同。 |
-| `ef` | 否 | `floor(1.5 × (top_k + offset))` | HNSW
图索引搜索时保留的候选宽度，必须为正整数。如果同时设置了 `refine_factor`，默认值为 `floor(1.5 × (top_k + offset)
× refine_factor)`。对非 HNSW 索引无效。 |
-| `use_index` | 否 | `true` | `true` 表示将兼容的物理 Lance Index Segment 规划为索引
Split，并将未覆盖的 Fragment 保留为 Flat Search Split；如果没有可用的兼容索引元数据，则退回按 Fragment
拆分。`false` 表示每个可见 Fragment 生成一个 Split，并强制执行 Flat Search。 |
-
-以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，如果向量列存在兼容索引，查询使用该索引创建时配置的
Metric；不存在兼容索引或 `"use_index" = "false"` 时，`uint8` 向量使用
`hamming`，其他当前支持的向量元素类型使用 `l2`。
-
-### 支持的向量索引类型
-
-当前内置 `lance-c v0.1.6` 明确支持以下 Lance 向量索引组合：
-
-| 索引类型 | 说明 | 主要查询参数 |
-|---|---|---|
-| `IVF_FLAT` | IVF 分区，分区内使用原始向量计算距离 | `nprobes` |
-| `IVF_SQ` | IVF 与 Scalar Quantization | `nprobes`、`refine_factor` |
-| `IVF_PQ` | IVF 与 Product Quantization | `nprobes`、`refine_factor` |
-| `IVF_HNSW_FLAT` | IVF 与 HNSW，图节点保存原始向量 | `nprobes`、`ef` |
-| `IVF_HNSW_SQ` | IVF、HNSW 与 Scalar Quantization |
`nprobes`、`ef`、`refine_factor` |
-| `IVF_HNSW_PQ` | IVF、HNSW 与 Product Quantization |
`nprobes`、`ef`、`refine_factor` |
-
-`vector_search()` 只负责查询，不负责在 Doris 中创建索引，也不提供指定索引类型或索引名称的参数。当 `use_index=true`
时，FE 从固定的数据集快照读取向量索引元数据，并选择与向量列和 Metric 兼容的一个逻辑索引；随后将该逻辑索引中仍覆盖可见数据的每个物理 Segment
分配给一个索引 Scan Split。每个索引 Split 都携带 Segment UUID 以及该 Segment 覆盖且在当前快照中可见的
Fragment，因此 BE 会检索指定的物理 Segment，而不是再次让 Lance 自行选择索引。
-
-一个 Lance 逻辑索引可以包含多个物理 Index Segment，一个物理 Segment 也可以覆盖多个 Fragment。未被所选索引覆盖的
Fragment 不会被遗漏：Doris 会为每个这样的 Fragment 增加一个执行 Flat Search 的回退 Split。如果 FE
无法生成可用的 Index Segment 计划，则退回按 Fragment 拆分。当 `use_index=false` 时，Doris
跳过索引元数据规划，并对每个可见 Fragment 强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance
内直接读取并比较向量。
-
-### Prefilter 和 Post-filter
-
-TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个搜索 Split 的 Lance Scanner，Lance
在 ANN 或 Flat Search 生成候选之前执行过滤：
-
-```sql
-SELECT user_id, category, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10",
- "filter" = "category = 'book'"
-)
-ORDER BY _distance ASC, user_id;
-```
-
-`filter` 中引用的列由 Lance 内部读取并计算；如果该列没有被 `SELECT` 或其他 Doris 表达式引用，它不需要作为列返回给
Doris。
-
-外层 `WHERE` 是 **Post-filter**。优化器会将它下移到 Doris 的 Lance Scan 中，但不会把它转换成 Lance 的
Prefilter。它的执行位置是：Lance 为每个搜索 Split 生成候选之后、Doris 执行局部和全局 TopN 之前。
-
-```sql
-SELECT user_id, category, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10"
-)
-WHERE category = 'book'
-ORDER BY _distance ASC, user_id;
-```
-
-因此，外层 `WHERE` 只过滤已经生成的候选，不会触发 Lance 补充候选，最终结果可能少于
`top_k`。如果过滤条件应该缩小向量候选的搜索空间并保证在过滤后的数据中选择最近邻，应使用 TVF 的 `filter` 参数。
-
-### 当前执行方式
-
-`vector_search()` 使用分布式候选搜索，而不是由一个 Scanner 扫描整个数据集。Split 的边界取决于索引覆盖范围：
-
-1. FE 在规划阶段固定一个正数版本的 Lance 数据集快照，并读取该快照中可见的 Fragment；当 `use_index=true`
时，还会读取向量索引元数据。
-2. 如果存在具有可用 Segment 覆盖信息的兼容逻辑向量索引，每个仍覆盖可见 Fragment 的物理 Index Segment 都会生成一个索引
Scan Split。该 Split 包含 Segment UUID，以及其 Fragment Bitmap 与固定快照中可见 Fragment
的交集，因此一个 Split 可以包含多个 Fragment ID。
-3. 没有被这些索引 Split 覆盖的每个可见 Fragment，都会生成一个独立的回退 Fragment
Split。这样，即使数据是在索引创建后追加的、尚未执行索引优化，也仍然可以被检索。如果不存在可用的 Index Segment 计划，所有可见
Fragment 都按 Fragment 拆分；当 `use_index=false` 时，所有可见 Fragment 直接使用 Flat Search
Split。
-4. 假设查询参数为 `top_k=K`、`offset=n`，每个索引或回退 Split 都请求最多 `K+n` 个候选，并且不在 Split 内应用
offset。索引 Split 只检索为其分配的物理 Index Segment，回退 Fragment Split 对自身 Fragment 执行 Flat
Search。TVF 的 `filter` 在候选生成前由 Lance 执行，外层 `WHERE` 则在候选生成后由 Doris Scan 执行。
-5. Doris 对所有 Split 返回的候选执行局部 TopN、Exchange 和全局 TopN，按 `_distance ASC` 合并；只有全局
TopN 应用 `offset=n`，跳过前 `n` 行后返回 `K` 行。
-
-因此，Split 级候选集只用于向全局合并提供候选，不能直接视为最终结果。索引 Segment Split、回退 Fragment Split 以及后续按
Row ID 取列都使用同一个固定快照。刷新索引覆盖会改变新追加 Fragment 的检索方式，但未被索引覆盖的 Fragment 仍会通过 Flat
Search 进入检索范围。
-
-执行顺序可以概括为：
-
-```text
-固定数据集快照
- -> FE Split 规划
- -> 索引覆盖：每个物理 Index Segment 一个 Split -> ANN Search
- -> 未覆盖或无索引数据：每个 Fragment 一个 Split -> Flat Search
- -> 每个 Split：Lance Prefilter -> ANN/Flat Search -> 最多 K+n 个候选
- -> Doris Scan Post-filter
- -> Doris 局部 TopN
- -> Exchange
- -> Doris 全局 TopN（应用 offset=n 和 limit=K）
- -> 可选的延迟物化 Fetch
-```
-
-### TopN 两阶段读取和延迟物化
-
-当 `experimental_topn_lazy_materialization_threshold` 大于 `0`、`top_k`
不超过该阈值，并且存在可以延迟读取的顶层列时，`vector_search()` 可以使用两阶段读取。默认阈值为 `1024`。第一阶段只传递完成候选过滤和
TopN 所必需的列以及内部 Row Location；全局 TopN 完成后，第二阶段只为最终保留的行读取其他输出列。
-
-例如，源表包含以下列：
-
-| 列 | 用途 |
-|---|---|
-| `user_id` | 最终输出列 |
-| `category` | 外层 `WHERE` 的 Post-filter 列 |
-| `title`、`payload` | 最终输出列 |
-| `embedding` | Lance 向量搜索列 |
-
-执行以下查询，其中 `K=10`、`n=3`：
-
-```sql
-SET experimental_topn_lazy_materialization_threshold = 1024;
-
-SELECT user_id, title, payload, _distance
-FROM vector_search(
- "table" = "lance_catalog.default.items",
- "column" = "embedding",
- "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
- "top_k" = "10",
- "offset" = "3"
-)
-WHERE category = 'book';
-```
-
-典型的两阶段列流转如下：
-
-| 阶段或算子 | 读取或输出的列 | 说明 |
-|---|---|---|
-| Lance Split Search | 内部使用 `embedding`；向 Doris 返回 `_distance`、`category` 和内部
Lance Row ID | `embedding` 用于 ANN/Flat Search，但没有被 SQL 投影时不作为结果列返回。每个 Index
Segment 或回退 Fragment Split 最多产生 `K+n` 个候选。 |
-| Doris Scan Post-filter | `_distance`、`category`、内部 Row Location | 执行
`category = 'book'`。外层 `WHERE` 的列必须留在第一阶段。Doris 将 Lance Row ID 和数据集映射编码为内部 Row
Location，Fetch 再通过该映射解析到同一个固定快照。 |
-| 局部和全局 TopN | 第一阶段必需列和内部 Row Location | 全局 TopN 按 `_distance` 合并，应用
`offset=n` 和 `limit=K`。 |
-| Row ID Fetch | 使用内部 Row Location 读取 `user_id`、`title`、`payload` | 对全局 TopN
保留的行，在同一个 Lance 数据集快照上调用 Row-ID 随机读取，不重新扫描 Fragment。 |
-| 最终 Materialize | `user_id`、`title`、`payload`、`_distance` |
将延迟列与第一阶段保留的列合并，形成 SQL 最终输出。 |
-
-第一阶段的必需列不只包括 `_distance` 和 Post-filter 列。凡是在全局 TopN 完成前被 Doris
表达式或算子引用的列，都属于第一阶段列。例如，如果查询增加 `ORDER BY _distance, user_id`，`user_id`
也需要提前读取，不能再等到第二阶段 Fetch。嵌套子列投影当前也不会延迟到 Row-ID Fetch。相反，只被最终投影使用的顶层列可以在第二阶段读取。
-
-TVF `filter` 中引用的 Prefilter 列与外层 `WHERE` 列不同：前者由 Lance 在搜索内部使用，并不因为出现在
`filter` 字符串中就必须返回到 Doris；后者由 Doris Scan 执行，所以必须进入第一阶段。
-
-将 `experimental_topn_lazy_materialization_threshold` 设置为 `-1` 会关闭两阶段读取。如果
`top_k` 大于阈值，或者没有可延迟的列，也会使用单阶段读取。单阶段模式会在 Scan 阶段返回查询所需的全部输出列，但向量搜索仍然按 Index
Segment 或回退 Fragment Split 并行生成候选，并由 Doris 合并全局 TopN；它不会因此退化为 Doris
对整张表做普通全列扫描。索引 Split 使用为其分配的物理 Index Segment，强制或回退到 Flat Search 时才直接比较向量。
-
-## 当前限制和建议
-
-- Lance Catalog 和 Lance TVF 当前仅支持读取，不支持 `CREATE
TABLE`、`INSERT`、`UPDATE`、`DELETE`、`TRUNCATE TABLE` 或写回 Lance。
-- 查询总是读取规划时选择的当前版本，不支持通过 SQL 指定 Version 或执行 Time Travel。
-- 使用不支持的列类型时，建议显式列出需要读取的列，避免 `SELECT *` 投影到不支持的列。
-- 对普通扫描使用 `EXPLAIN` 检查 `lancePushdownPredicate`，确认目标条件是否已下推。
-- 向量检索前应在 Lance 中创建与查询方式匹配的索引；小数据集或验证场景可以设置 `"use_index" = "false"` 使用 Flat
Search。
-- 向量查询需要稳定顺序时，显式使用 `ORDER BY _distance ASC` 并增加唯一 Tie-breaker。
-- 需要在向量候选生成前过滤时使用 `vector_search()` 的 `filter`；外层 `WHERE` 只过滤每个搜索 Split
已生成的候选，并在 Doris 全局 TopN 之前执行，应允许其最终结果少于 `top_k`。
-- 使用 `EXPLAIN` 检查 `lanceSearchFragments` 和
`lanceSearchIndexSegments`。前者表示固定快照中的可见 Fragment 数量，后者表示 FE 选择的物理 Index Segment
Split 数量；此外还可能存在回退 Fragment Split。
diff --git a/sidebars.ts b/sidebars.ts
index 4cdb2e89cdd..6fa0fe08e7c 100644
--- a/sidebars.ts
+++ b/sidebars.ts
@@ -661,7 +661,6 @@ const sidebars: SidebarsConfig = {
'lakehouse/best-practices/doris-dlf-iceberg',
],
},
- 'lakehouse/catalogs/lance-catalog',
{
type: 'category',
label: 'Paimon Catalog',
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
