u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-6163872351
This supersedes the v5.1 proposal following the synchronous-direction review. The baseline facts — the commit-resolved pinned revisions, panic behavior at the FFI boundary, the binding entry points, and the Rust-to-C error mapping — were re-verified against the pinned sources, and every execution parameter the direction comment left open was resolved to the reading most consistent with it. Section 11 records that evidence and the single remaining decision (client-disconnect semantics), and asks for confirmation of the 4.2 boundary. # Design Proposal v6 — Synchronous Lance Index Lifecycle for Doris 4.2 > - **Prepared for publication:** 2026-10-08 (Asia/Shanghai); revised 2026-10-09 > - **Target:** Apache Doris `branch-4.1` / Doris 4.2 > - **Issue:** [apache/doris#66497](https://github.com/apache/doris/issues/66497) > - **Status:** v6 publication candidate (revised 2026-10-09 after local verification of the pinned ABI facts and the failure-contract review); supersedes v5.1 following [the synchronous-direction comment](https://github.com/apache/doris/issues/66497#issuecomment-6053021847) (2026-10-08) > - **Provenance:** v5/v5.1 remain issue-history provenance only; they provide no implementation guidance here ## 0. Authority and revision summary This standalone document defines the capability and safety contract for Lance index lifecycle management in Doris 4.2 under a **synchronous execution model**. If accepted, it supersedes the v5.1 minimal one-shot proposal. ### Context for this revision On 2026-10-08, after reconsidering the implementation against the execution model of Iceberg `rewrite_data_files`, the reviewer changed the previously accepted v5.1 direction: a durable asynchronous Lance index job framework is no longer a requirement for the initial implementation. The SQL statement itself validates the request, dispatches resource-isolated work to a BE, waits for completion, refreshes metadata after a confirmed commit, and returns the result. Internal parallel execution does not require an asynchronous SQL/job interface. The job infrastructure already introduced on `branch-4.1` — the durable job state machine, manager, persistent same-name fences, unresolved-job quotas, and job replay of #67235, plus the job-specific parts of #67630 (durable job creation and JobId results, `SHOW LANCE INDEX JOB(S)`, catalog guards tied to unresolved jobs) — is reverted explicitly rather than left disabled or maintained as a second execution path. The reusable privilege, schema, parameter, snapshot, and `IF`-condition validation, the read-only `SHOW INDEX`, and the physical-index inspection are preserved and adapted for synchronous execution. The worker resource protections — heavy Lance work outside FE and the BE main process, hard worker resource limits, bounded concurrency/queueing, deadlines, and reliable child cleanup — are independent of durable job management and remain requirements. Relative to v5.1, this revision retains: the SQL surface and type/property matrix, the authoritative one-snapshot metadata contract, admission-time authoritative preflight and `IF` behavior, dataset-version and schema-contract revalidation before the mutating native invocation, hard worker process isolation, typed provider-result classification, and secret handling. It removes: durable job records, the compact `PENDING -> RUNNING -> ...` lifecycle, same-name fences, unresolved-job quotas, `FORCE_RELEASE`, job inspection SQL, and replay/failover recovery. The `UNKNOWN` concept survives only as a **client-visible indeterminate-outcome error class**, never as a persisted state. ### Baseline The baseline is resolved at commit level, because package versions and release tags alone do not determine API or error semantics: Doris `branch-4.1` at `c8c0ddca4d`; FE Java binding `lance-core` `12.0.0`; BE C binding `lance-c` at commit `98468344` (its package version string still reads `0.1.9`), whose Cargo manifest pins the Rust core at `68c12dfd`. The BE binding is identified by the exact source URL recorded in `thirdparty/vars.sh` at the stated Doris commit — not by its version string or tag. Any future statement about "what the pinned bindings do" resolves the vars.sh-to-commit-to-manifest chain before asserting a fact. No 4.2 guarantee depends on upgrading or extending those APIs; Section 8 records which extension capabilities the pinned bindings do and do not expose. Normative terms use **MUST**, **MUST NOT**, **SHOULD**, and **MAY** in their ordinary requirements sense. Sections explicitly marked future work are non-normative for 4.2. > **Review focus:** Section 11 records the verified baseline evidence and the single open decision (client-disconnect semantics), and asks for confirmation of the 4.2 boundary. ## 1. Scope and release boundary ### 1.1 Core safety contract Doris 4.2 executes one Lance mutation inside a single synchronous SQL statement without retrying after dispatch may have occurred, fabricating an outcome, or implying that nothing committed when the outcome is uncertain. **Mutation outcome and metadata refresh are independent dimensions.** A mutation statement returns exactly one of four outcome classes: 1. **success** — including an authoritative-preflight no-op, which returns only after its required bounded cache invalidation completes (Section 6.4); a committed success is followed by the applicable refresh obligations; 2. **confirmed failure** — a complete trusted result proves this invocation did not commit; 3. **indeterminate outcome** — an explicit typed error stating the mutation may or may not have committed, directing the user to authoritative index metadata; 4. **committed, refresh incomplete** — a complete trusted success was received and the required refresh did not complete (Section 6.4); the build outcome is known and is never reported as a build failure. A client that cannot receive a result packet — a timeout with a lost response, a disconnect, or FE loss — treats the outcome as indeterminate. This is a documentation and driver rule (Sections 5.5 and 6.2), because no SQL error can be delivered on a closed connection. ### 1.2 Required in 4.2 The 4.2 deliverable includes: - Directory Catalog `CREATE INDEX`, `CREATE OR REPLACE INDEX` (full same-name rebuild), and `DROP INDEX` by logical name, for the initially verified vector and scalar index types, **executed synchronously within the statement**; - `SHOW INDEX` for logical index metadata and the `lance_index_entries` TVF for physical index UUID and dataset version (both unchanged from the merged read-only work); - basic `IF NOT EXISTS` and `IF EXISTS` behavior based on authoritative Lance metadata; - table-level `SHOW` and `ALTER` authorization; - admission validation preserved from the merged work: privilege, target, schema, parameter, snapshot, and `IF`-condition checks before any dispatch, with the final authoritative preflight inside the master's name entry (Section 4.3); - one resource-isolated worker process on one selected BE through a verified hard boundary: memory limit, PID/thread limit, wall-clock runtime limit, reliable termination detection and child reaping across the whole isolated execution domain; - bounded concurrency and bounded queueing for mutation statements, and one total time budget per statement covering admission reads, queueing, dispatch, execution, and refresh (Section 5.5); - worker capacity accounting held in memory by the BE supervisor until termination of the isolated execution domain is confirmed (Section 5.4); - typed result classification per Section 6, including the distinct indeterminate-outcome and committed, refresh incomplete error classes; - the refresh obligations of Section 6.4, with refresh failure reported distinctly from build failure; - focused crash, OOM, panic, privilege, fault-injection, and end-to-end tests; - user documentation for syntax, synchronous semantics, the four outcome classes, limitations, and inspection. Lance manifests remain the authoritative index metadata. Doris persists **no** mutation control state: no job record, no fence, no quota book, no replay. ### 1.3 Explicitly deferred from 4.2 These exclusions are normative and cannot gate 4.2: - persistent job history, FE failover recovery for in-flight mutations, resumable builds, and durable same-name serialization across failures; persistent recovery, if ever needed, is a separate issue that must justify itself rather than a prerequisite for index lifecycle support; - job inspection and release SQL and every persisted-`UNKNOWN` operational protocol — `SHOW LANCE INDEX JOB(S)`, `RESOLVE ... FORCE_RELEASE`, audited release notes, and late-commit warnings attached to records; - automatic `UNKNOWN` watchers or reconciliation daemons, idempotency protocols, automatic retry of dispatched mutations, incremental `BUILD INDEX`, coverage repair, build progress, and continuously appended-table freshness SLAs; - running-cancellation rollback: cancellation is best-effort termination only and never undoes a commit (Section 6.3); - distributed index construction as a 4.2 capability (Section 8 states the extension path; the initial version uses one worker); - REST inspection and mutation surfaces; - creating additional index types, composite indexes, or nullable indexed fields, and richer physical metadata (dropping existing composite, nested, or legacy indexes by name remains in scope); - provider-backed table/dataset incarnation registries, external same-URI replacement detection, normalization migration, and mixed-version fence protocols; - a duplicate logical TVF, server-side exact logical-index counts, and `countRows()` surfaces; - broad catalog-wide external-DDL transaction machinery beyond the in-memory guards of Section 4. ## 2. User-visible contract ### 2.1 SQL surface ```sql CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (embedding) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "l2", "num_partitions" = "256", "num_sub_vectors" = "16" ); CREATE INDEX idx_btree ON lance_ctl.db.tbl (event_time) USING BTREE; CREATE INDEX idx_bitmap ON lance_ctl.db.tbl (category) USING BITMAP; CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (embedding) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "cosine", "num_partitions" = "256", "num_sub_vectors" = "16" ); SHOW INDEX FROM lance_ctl.db.tbl; SELECT * FROM lance_index_entries("table" = "lance_ctl.db.tbl") WHERE IndexName = "idx"; DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl; ``` `USING ANN` is a neutral DDL category. It does not add or imply Doris internal ANN query syntax for Lance. Lance vector queries continue to use `vector_search()`. `CREATE OR REPLACE` maps to `replace=true` and fully rebuilds the logical name. It converges by name but is not idempotent: another invocation may produce a different UUID and dataset version. It is mutually exclusive with `IF NOT EXISTS`; `USING`, target column, and required properties are validated before dispatch. Only top-level CREATE, REPLACE, and DROP mutate Lance indexes in 4.2; `ALTER TABLE ... ADD/DROP INDEX` and `BUILD INDEX` remain unsupported. Internal tables are unchanged. ### 2.2 Synchronous execution semantics All Lance index mutations are forwarded to and executed on the active master FE. An admitted mutation executes within the statement and returns when one of the Section 1.1 outcome classes is known: ```text static validation (any FE) -> forward to master; forwarding preserves single-send (Section 4.3) -> acquire the in-memory active-name entry; same-name busy fails fast -> final authoritative IF preflight, inside the entry (Sections 2.3, 4) -> bounded admission: concurrency limits, finite queue, total budget check (Section 5.5) -> select one BE; the BE supervisor reserves worker capacity (Section 5.4) -> single dispatch request carrying invocation identity, remaining budget, and its verifiable expiration -> BE supervisor launches the resource-isolated worker -> one pinned lance-c mutating invocation against the admitted dataset version -> complete typed result classified (Section 6.5) -> refresh obligations (Section 6.4) -> release worker capacity and the name entry on confirmed termination of the execution domain -> return to the client ``` The statement returns no job id and exposes no job surface. A client that loses the connection, times out, or kills the statement follows Sections 5.5 and 6.2 and MUST NOT blindly resubmit. ### 2.3 Submission and `IF` behavior Admission reads authoritative metadata before any dispatch: - plain CREATE fails if the normalized logical name already exists — existence is determined from the physical list together with available logical descriptions, so a legacy index omitted from logical descriptions still occupies its name (Section 3.1); - CREATE IF NOT EXISTS returns an immediate no-op only when the existing authoritative normalized name, indexed columns, physical family, and algorithm match. A property is compared only when the same bounded logical metadata already used by `SHOW INDEX` exposes a stable value; the exposed mappings are `metric` against `metric_type`, `num_sub_vectors` against `compression.num_sub_vectors`, and the effective `num_bits=8` against `compression.num_bits`. `num_partitions` is not reliably exposed and is therefore not compared, and no substitute value (for example `target_partition_size`) may stand in for it. No additional canonicalization or metadata read is a prerequisite; - CREATE IF NOT EXISTS fails on an authoritative mismatch rather than treating a different same-name definition as success; - DROP fails if the authoritative name is absent; - DROP IF EXISTS returns an immediate no-op when the authoritative name is absent; - REPLACE requires an existing or creatable unambiguous target name and always represents a new one-shot mutation when admitted. Two ordering rules close the preflight races. First, **the final authoritative preflight runs after the active-name entry is acquired on the master**, immediately before dispatch admission: a preflight taken earlier — before forwarding, before queueing, or before the entry — is not sufficient to dispatch, so a request that queued behind another same-name statement cannot proceed on a stale snapshot. Second, an authoritative no-op completes its required bounded cache invalidation before returning (Section 6.4), so a stale cache cannot contradict the returned state; the invalidation closure and its residual failure class are defined there. Authoritative preflight is not an external lock. A post-preflight race against an external writer is classified from the result rules in Section 6.5; Doris never turns a generic post-invocation error into an IF no-op by inspecting message text. ### 2.4 Supported type and property matrix | Type | Indexed-column contract | User properties and checks | | --- | --- | --- | | `IVF_PQ` | Exactly one non-null fixed-size-list field with `FLOAT16` or `FLOAT32` elements | `index_type=IVF_PQ`; `metric` is `l2`, `cosine`, or `dot`; positive required `num_partitions` and `num_sub_vectors`; subvectors divide the vector dimension; configured static bounds apply; `num_bits=8` is fixed | | `BTREE` | Exactly one non-null scalar field from the pinned predicate-pushdown set: integral, floating, decimal, string, date, or timestamp | No user build properties; C `params_json=NULL` | | `BITMAP` | Exactly one non-null boolean, integral, string, or date field | No user build properties; C `params_json=NULL` | For CREATE and REPLACE, Hamming distance, nullable/unsupported nested fields, `FLOAT64`, `UINT8`, or `INT8` vector elements, composite indexes, arbitrary scalar JSON, and unknown properties or values fail before dispatch — these admission limits never restrict what can be dropped by name. FE enforces static syntax, type, and bounds; the worker repeats the independently checkable facts against the admitted snapshot before the mutating native invocation. Vector element nullability is not a schema-level admission check: the manifest has no slot for element nullability, and the pinned Java SDK reconstructs every fixed-size-list element as nullable regardless of what was written, so element nullability is unobservable at the schema level. The two validation layers are kept separate: facts independently checkable before the call — the element type and the non-null top-level field — are checked before the mutating native invocation; validation of actual element values happens inside the provider's build, within the worker's budget, and its failures classify under the post-invocation rules of Section 6.5, never as provable pre-invocation failures. ## 3. Authoritative metadata contract ### 3.1 One pinned Directory snapshot Every SHOW, physical inspection, IF check, and post-commit refresh uses one opened Directory `Dataset` snapshot. One snapshot does not mean one native call: the read chain issues several calls against the same opened dataset and keeps one overall deadline for the chain. - raw index names come from the index listing; reserved system-entry names (`__lance_frag_reuse`, `__lance_mem_wal`) are filtered and the raw list is deduplicated by exact name; - logical descriptions are fetched per name with an exact-name criteria — a criteria-free bulk description is not used, because converting the details of legitimate system entries can fail the whole call before Doris filters anything. A per-name lookup returns exactly one non-null description carrying the requested name, or — only in the legacy case below, where every physical segment of that index lacks readable details — an empty result that leaves the name to the physical list; a null, duplicate, or wrong-name result fails closed; - the same snapshot schema resolves indexed field IDs; - the physical listing supplies physical UUIDs and dataset versions. The chain bounds raw entries and logical names separately, both finite. **Name occupancy for mutation is determined from the physical list together with the available logical descriptions.** A legacy index whose segments all lack readable details may be omitted from logical descriptions while its name still appears in the physical list: plain CREATE rejects such a name as occupied, CREATE IF NOT EXISTS does not claim a match without a reliable definition, and DROP IF EXISTS does not claim absence from an incomplete logical list alone. Missing legacy details are not malformed metadata; malformed or oversized data still fails closed with a typed bounded error, and the two cases are never conflated. Whether SHOW continues to omit such legacy logical descriptions is a fixed, documented behavior, not an accident of the read path. Normal inspection and FE admission MUST NOT call `Dataset.countRows()`. `Dataset.getIndexStatistics()` is also forbidden because the pinned version may migrate metadata and write a manifest. Provider-local build validation stays in the isolated worker and creates no Doris row-count surface. FE metadata reads have bounded results and deadlines through the existing bounded inspection executor (`LanceIndexInspectionExecutor`, 4 concurrent / 16 queued on the baseline), which remains the read-side owner: admission reads go through it, and a statement exiting does not reclaim capacity of a read still inside JNI. Timeout does not imply JNI cancellation; ownership remains until the native call returns. A separate read helper process is not a 4.2 requirement. ### 3.2 Logical `SHOW INDEX` `SHOW INDEX` keeps the existing MySQL-compatible 13-column schema and returns one row per logical index/column. For Lance: - `Key_name` is the exact display name; - `Column_name` comes from the same-snapshot field-ID resolution; - `Index_type` is the logical Lance algorithm such as `IVF_PQ`, `BTREE`, or `BITMAP`, not the SQL category `ANN`; - `Properties` is deterministic, valid, allowlisted JSON under a reviewed finite UTF-8 bound. There is no second logical TVF or server-side exact-count API; callers may count distinct `Key_name` values in the bounded SHOW result. ### 3.3 Minimal physical inspection `lance_index_entries("table"="ctl.db.tbl")` returns one row per physical entry: `CatalogName, DatabaseName, TableName, IndexName, IndexUuid, DatasetVersion`. The TVF exists only because UUID and dataset version do not fit the established SHOW schema. It exposes no fragments, coverage, provider/consistency state, opaque details, or transaction properties. The TVF requires one table argument, table `SHOW`, and an optional ordinary predicate over the bounded result. It is all-or-error and never silently truncated. ### 3.4 Pinned type vocabulary, system entries, and bounds The pinned Java SDK has different logical and physical vocabularies: - per-name logical descriptions derive a concrete vector algorithm such as `IVF_PQ`, while the physical listing may expose the umbrella manifest type `VECTOR`; - scalar values differ in case and separators, for example `BTREE` versus `BTree` and `LABEL_LIST` versus `LabelList`. Authoritative IF and internal joins use one normalization rule: - normalize case and ignore underscores for family comparison; - accept physical `VECTOR` only for supported vector algorithms; - accept physical `SCALAR` only for supported scalar algorithms; - never match an internal Doris index family to a Lance family. Logical SHOW displays the concrete per-name description value. The minimal physical TVF does not expose the umbrella type. Duplicate logical names or UUID ownership, unknown field IDs, malformed or oversized data, and required logical/physical mismatches fail closed with a typed bounded error. Directory inspection caps logical names, physical entries, external strings, fields per index, and aggregate field-name bytes. Exact values follow Doris configuration conventions. Because these post-JNI checks do not bound native allocation, focused FE stress/failure tests cover accepted and over-limit data. ## 4. Target identity, admission, and Doris-side guards ### 4.1 Stable target and logical name The target key used for validation, DDL guards, and in-memory concurrency is: ```text ( persisted catalog identity, provider = DIRECTORY, normalized stable dataset locator, normalized logical-index-name bytes ) ``` The catalog identity is persisted configuration. The request's display name and normalized name bytes live only in bounded invocation memory for the duration of the statement and its cleanup — this contract adds no mutation persistence. Normalization v1 is the UTF-8 result of Java `toLowerCase(Locale.ROOT)`. Doris preserves display case, rejects new case-only duplicates, resolves a unique match to its stored name, and fails mutation on ambiguous external case-only collisions. There is no durable fence keyed on this tuple. Credential-bearing URLs are never identity; URI aliases are external writers. Replacing the dataset at the same URI concurrently with a mutation is unsupported and lands in the indeterminate class of Section 6. ### 4.2 Dataset-version and schema-contract revalidation CREATE and REPLACE admission records the exact selected/latest dataset version and the following ordered contract: ```text schema_contract_version = 1 indexed_field = { field_id, normalized_name, normalized_type, nullable, fixed_size_list_dimension, // when applicable vector_element_type, // when applicable vector_element_nullable // when applicable } ``` `normalized_type` includes relevant parameters such as decimal precision/scale and timestamp unit/time-zone semantics. Unindexed fields are excluded. Before the mutating native invocation, the worker reopens the admitted dataset version, independently recomputes contract v1, and compares the ordered representation. An unavailable version, mismatch, or unsupported contract produces a complete pre-invocation **confirmed failure** with typed `STALE_ADMISSION` or `UNSUPPORTED_SCHEMA_CONTRACT`; the worker never switches to latest. This revalidation confirms the admission snapshot itself; it does not observe whether storage-latest has since advanced, and any post-admission race is arbitrated by the provider's commit machinery and classified under Sections 4.3 and 6.5. No additional worker-start latest-equality gate is added: this design does not require storage-latest to still equal the admitted version when the worker starts, and such a gate would turn concurrent appends that the provider can safely arbitrate into early rejections — Section 4.3 promises no such failure — while still not guaranteeing equality at final commit. The worker may still read storage-latest for result diagnostics; such reads never gate the invocation. Java admission and Rust execution share golden contract fixtures. A DROP target is revalidated against the pinned admission snapshot: the worker reopens the admitted dataset version and resolves the normalized name uniquely to a stored name in that snapshot's bounded physical list, then issues one drop by that name. This confirms the admission snapshot only — it does not prove that storage-latest has not advanced since admission, and a concurrent external change is arbitrated by the provider's commit machinery and classified under Sections 4.3 and 6.5. No field-schema contract is built for DROP — CREATE's single-column and algorithm restrictions never limit what can be dropped, and composite, nested, or legacy indexes whose names appear in the physical list are droppable by logical name. A same-URI dataset replacement that reproduces the same version cannot be detected by this revalidation; no indeterminate-detection promise is made for it. ### 4.3 Same-name concurrency and admission serialization With fences and quotas removed, same-name concurrency is handled by master-local admission plus provider-side arbitration, and nothing else: - **all mutations execute on the active master FE.** Statements arriving on a follower are forwarded, consistent with existing DDL forwarding conventions. Single-send must hold across every hop — follower-to-master forwarding, master-to-BE dispatch, and thrift reconnect or timeout paths — so that no path can produce a second execute request for one statement; - the master holds an **in-memory active-name entry** per target logical name. A same-name request arriving while an entry is active fails fast with a typed busy error; it does not queue behind the running statement and does not block a worker thread waiting for a lock; - **the final authoritative preflight runs after the entry is acquired** (Section 2.3), so a racing or queued request cannot dispatch against a stale preflight snapshot; - the entry is retained until termination of the isolated execution domain is confirmed — it outlives statement return, statement timeout, and client disconnect, because the worker may still be running. A later same-name request may therefore see busy until cleanup completes; releasing the client connection never releases the entry by itself. When the termination proof cannot be delivered or re-acquired (Section 5.3), busy persists until the coordinator epoch is rebuilt; the entry is bounded by the finite epoch cleanup total (Section 5.4) and is never released by a TTL or heartbeat guess; - **Lance-side arbitration is a separate mechanism, not a classification promise.** The provider's manifest arbitration may resolve concurrent commits internally (including rebase against concurrent appends). Doris does not rely on receiving a typed conflict: distinct provider conflict variants share the aggregate internal error code in the pinned C ABI, so a lost same-name race may surface as a generic internal error and classifies as **indeterminate** under Section 6.5. Pinning the build snapshot does not promise that the committed version equals the admission version plus one, and does not promise failure on concurrent append; - no entry survives an FE restart. Doris provides no durable same-name serialization across failover or against external writers; this window is an explicitly accepted exposure, and observing a same-name index never attributes creation (Section 6.2). ### 4.4 Credentials No component persists credentials: dispatch, authoritative reads, and refresh resolve current catalog credentials when executed. A credential-only catalog ALTER is not target-changing and MUST remain allowed; provider, stable-locator, and selected-version changes remain guarded (Section 4.5). Known-expired credentials fail before the mutating invocation as a confirmed failure. No credential-TTL or safety-margin protocol is required. Section 7 defines secret handling. ### 4.5 Minimal DDL guard boundary While a mutation executes, an **in-memory active-target token** on the master guards target-changing DDL: a Doris DDL that changes the persisted catalog identity, Directory provider, normalized locator, or selected-version semantics fails fast with a busy error. Credential-only ALTER, ordinary cache refresh, unrelated properties, and external appends remain allowed. No broad catalog lock is held across execution. Three distinct owners are involved and never conflated: the per-name admission entry (Section 4.3), the short FE metadata/DDL locks held only during bounded validation and admission (Section 5.1), and the active-target token (this section). FE restart loses the token; that window is accepted and is not serialized across failures. Target-aware routing precedes internal `catalog.Index` translation. Internal validation/serialization is unchanged, and Lance algorithms do not extend the persisted internal `IndexType` enum. ## 5. Synchronous execution and resource boundary ### 5.1 Execution flow The execution pipeline is the one stated in Section 2.2; this section fixes its locking and resource boundaries. FE metadata/DDL locks cover only bounded validation and admission — not dispatch, native execution, waiting, metadata reads, or refresh. ### 5.2 Hard process isolation The pinned C binding catches Rust panics at the FFI boundary and reports them as a typed panic error, and its build configuration rejects unwinding-incompatible abort settings at compile time. Nevertheless OOM, stack overflow, and double panics can still terminate a process, and the native runtime creates threads outside Doris `MemTracker` control. Production mutation therefore never runs inside FE, the BE main process, or an ordinary RPC handler. Before Lance/Rust initializes materially, the selected BE launches a same-build worker in a verified OS-enforced boundary providing: - a hard memory limit; - a hard PID/thread limit; - a wall-clock runtime limit; - reliable termination detection and child reaping across the whole isolated execution domain. A deployment unable to verify the boundary rejects before the mutating invocation; there is no in-process or soft-RSS fallback. Launcher, containment, environment, diagnostic, and handshake details are not public contracts. The BE supervisor and its queues are bounded. Capacity reservation and BE selection are strictly non-invoking: they do not send the execute request, launch a worker, initialize Lance/Rust, or make native calls. Once the single execute request is sent, a complete trusted identity-matched busy or other pre-invocation rejection is a terminal **confirmed failure** and is never automatically retried within the same statement. Local/`file://` mutation has a separate disabled-by-default operator assertion, requires exactly one FE and BE, normalized absolute paths, pre/post version and identity checks, and rejection on topology change before dispatch. These checks cannot prove a shared mount namespace; object storage is the production mode. ### 5.3 Invocation identity, result envelope, and termination evidence - FE records an immutable invocation identity (invocation id, selected BE process epoch, budget) before sending the single execute request, and every response path is matched against it; - the worker/supervisor result envelope contains only what is needed to classify the one invocation: the matching invocation identity and BE epoch, complete trusted pre-invocation-failure and native-return markers, the saved `LanceErrorCode` read before the consuming error message, a bounded sanitized message, and matching domain-termination proof when available; - only a complete identity-matched result proves **confirmed success** or **confirmed failure**; a missing marker never proves the mutating native call was not entered. EOF, signal, timeout, OOM, BE loss, malformed/partial protocol, or identity mismatch after dispatch yields the **indeterminate** class; - the adapter validates deterministic user errors before the mutating native invocation and classifies only saved typed codes, never message text. It reads the code before the consuming message, checks it before treating a zero count as empty, and releases returned strings with `lance_free_string`; - **termination proof covers the whole isolated execution domain, not only the direct child**: capacity is not released while surviving descendants may exist. A supervisor-level check that the containment holds no remaining processes may serve as the domain proof; - **a complete result and a termination proof are two different kinds of evidence.** A received, identity-matched result finally classifies the mutation and is never rewritten by a later termination proof. A termination proof arriving after a lost result only releases resources; it never converts an indeterminate outcome into an attributed one; - **termination proof is handled separately from the mutation result.** When a result or termination report is lost in transit, the current invocation's in-memory cleanup channel may re-acquire or confirm the proof — a bounded acknowledgment or a read-only status query are implementation details. The cleanup channel never re-sends the execute request, and a proof acquired this way never rewrites an indeterminate outcome; - **when no proof can be obtained, admission stays fail closed.** A same-name request may observe busy until the coordinator epoch is rebuilt, not merely until the worker happens to end. Paths that provably never dispatched, or that carry complete never-launched evidence, have independent release conditions and do not wait on a child that does not exist. ### 5.4 Capacity ownership and queueing - **actual worker capacity is owned by the BE supervisor.** A slot is reserved at dispatch and released only when termination of the whole isolated execution domain is confirmed (Section 5.3); the direction comment states this requirement outright — worker capacity must remain accounted for until termination is confirmed. The BE does not release capacity because the FE disappeared or a client connection ended. A BE restart may only assume old executors are gone when they actually terminated under the old containment, or when OS protections guarantee they die with the old supervisor; heartbeat timeout or address re-registration alone does not prove the old process epoch is dead; - **FE-side limits are admission limits for the current coordinator epoch.** The master enforces configurable concurrent-statement limits (cluster-wide and per catalog) in memory, with one finite admission queue; same-name busy or a full queue returns a typed busy error instead of an unbounded wait. Retained name entries, active-target tokens, and cleanup contexts that outlive their statement count against a finite current-epoch cleanup total — statement return never bypasses that accounting, so repeated statements against different names cannot accumulate unbounded historical entries; the sum of BE capacities does not bound these FE-side entries. These limits do not pretend to be a cross-restart hard worker cap, and ending a statement or connection does not release capacity belonging to a still-running worker; - **the cluster-wide hard bound on simultaneously live workers is the sum of the finite per-BE supervisor capacities**, independent of FE session lifecycle. Topology growth or configuration changes do not turn a smaller FE-side global parameter into that sum. A precise per-catalog hard quota that survives failover would need a separate capacity-allocation or BE-side aggregation mechanism and is out of scope; - **queueing is two bounded tiers**: the finite master admission queue in front of dispatch, plus a bounded receive/worker queue on each BE as a defense line. A dequeued request revalidates the target, the preflight, and its remaining budget before dispatch; time spent in either queue counts into the statement budget (Section 5.5). No cross-BE work stealing in 4.2. ### 5.5 Time budget, cancellation, and disconnect - **one total time budget** — the direct operationalization of the direction comment's wait-refresh-return flow — starts at statement entry, before the admission reads, and covers admission reads, queueing, dispatch, execution, and refresh, with a termination and refresh margin reserved. The execute request carries the remaining budget with verifiable expiration information under a declared cross-host clock-skew assumption (or a verified transit-time upper bound conservatively deducted); the BE rejects an already-expired request at receipt and again at dequeue, before the mutating native invocation, and converts the admitted remainder into a monotonic local runtime limit independent of FE connectivity — not reset by BE reselection, queueing, or reconnection. Follower-to-master forwarding carries the same original budget. The mutation budget is its own configured bound: a generic query timeout is not assumed to suffice, and insufficient configuration surfaces as rejection, no t silent overrun; - a request dequeued with insufficient remaining budget is rejected **before the mutating native invocation** as a confirmed failure; - statement timeout and explicit cancellation request **best-effort termination** of the worker. Best-effort termination does not mean reaping completes before the SQL returns: capacity and the active-name entry wait for termination evidence (Sections 5.3, 5.4); - **a complete trusted success already received is never rewritten by a later timeout or cancellation.** If the refresh cannot complete within the budget, the statement returns **committed, refresh incomplete** (Section 6.4). Only the absence of a reliable mutation outcome yields indeterminate; - **closing a socket cannot deliver any SQL error.** A client that disconnects — voluntarily, by timeout with a lost response, or through FE loss — can receive no result packet and MUST treat the outcome as indeterminate and consult authoritative metadata. This is a documentation and driver rule; - **disconnect after dispatch:** the dispatched worker may run to its own completion or deadline within its budget; a live FE keeps a bounded completion, refresh, and cleanup continuation. This neither guarantees a commit nor releases capacity. Explicit KILL or statement timeout, by contrast, requests best-effort termination. ## 6. Failure contract ### 6.1 Promise and non-goals The initial version does not promise persistent job history, FE failover recovery, resumable builds, or durable same-name serialization across failures. All in-flight control state is in memory. An FE restart or failover during a statement is observed by the client as a disconnect and follows Sections 5.5 and 6.2. ### 6.2 Indeterminate outcomes A timeout, disconnect, worker loss, or lost response after dispatch, when no complete trusted outcome has been received, MUST NOT be presented as proof that nothing committed. Such a statement returns an explicit **indeterminate-outcome error** that: - states that the mutation may or may not have committed; - is never followed by an automatic retry of the potentially committed mutation — neither by Doris nor recommended to the client as a blind resubmission; - directs the user to inspect authoritative index metadata: `SHOW INDEX FROM ...` and `lance_index_entries(...)`; - never attributes observed external state to this request: observing a same-name index does not prove which request created it. ### 6.3 Cancellation Cancellation (query kill, statement timeout) triggers best-effort worker termination and returns control to the client; it cannot undo an existing commit. If a complete trusted success was already received before cancellation, the statement reports the committed, refresh incomplete class rather than indeterminate (Sections 1.1 and 5.5). Worker capacity and the active-name entry remain held until termination is confirmed (Sections 5.3, 5.4). ### 6.4 Refresh obligations independent of outcome Mutation outcome and metadata refresh are independent. One bounded refresh action runs inside the statement where the table below requires it; there is no refresh job and no background retry framework. Refresh failure keeps the mutation classification and attaches a separate refresh diagnostic. | Fact already known at classification | Mutation attribution | Refresh responsibility | | --- | --- | --- | | complete matching native success | committed by this invocation | refresh before returning success | | preflight `IF` no-op — authoritative state already satisfies the condition | no mutation executed this statement | complete the required bounded invalidation-and-refresh initiation on the executing FE before returning the no-op (below); an optional authoritative reload failing afterwards is an independent diagnostic only | | complete trusted commit-conflict result (saved typed code) | this invocation did not commit | refresh the externally advanced metadata | | pre-invocation revalidation observed relevant version/metadata advancement | this invocation did not commit | refresh the observed advancement; no forced additional reads without evidence | | post-invocation aggregate error or lost communication | unknown | refresh never attributes; the user may explicitly refresh or inspect | A confirmed commit followed by a refresh failure MUST be reported as a **distinct typed error — committed, refresh incomplete** — never as a build failure; the direction comment requires exactly this distinction. The commit is a fact, Lance metadata is authoritative, and the user may retry inspection or refresh through ordinary means. The build outcome is never rewritten by refresh failure. The no-op branch closes the same way. Its required actions are a bounded **local cache invalidation on the executing (master) FE** and **initiation of the ordinary external-table refresh propagation** — the existing edit-log-and-replay mechanism every other external refresh already uses — both retried inside the statement and practically always completable. Reads served by the executing FE then re-resolve from authoritative metadata and never hit cache entries predating the invalidation; caches on other FEs converge through the existing refresh-propagation semantics, and this contract claims no new cross-FE freshness guarantee beyond that existing mechanism. An optional authoritative metadata reload that fails afterwards attaches an independent diagnostic and does not change the classification. In the residual pathological case where the invalidation or the propagation initiation cannot complete, the statement returns a typed refresh error in the **confirmed-failure family**: no mutation was executed, it is not a build failure, and it is neither committed nor indeterminate. A native success proves that this invocation committed. It does not guarantee that another external writer has not since replaced or removed the index; "the index exists" is a statement about this commit, not a promise about the present external state. ### 6.5 Provider-result classification | Complete saved result | CREATE | REPLACE | DROP | | --- | --- | --- | --- | | rejection before the mutating native invocation, including version/schema/credential/resource revalidation | failure | failure | failure | | `LANCE_OK` from the one native mutating invocation | success (+refresh) | success (+refresh) | success (+refresh) | | complete trusted saved `LANCE_ERR_COMMIT_CONFLICT` (exact code) | failure (+refresh) | failure (+refresh) | failure (+refresh) | | any other saved typed code after the mutating invocation began — including generic `NOT_FOUND`, `INVALID_ARGUMENT`, `NOT_SUPPORTED`, `INDEX`, `IO`, aggregate `INTERNAL`, and typed panic | indeterminate | indeterminate | indeterminate | | no matching complete response, signal, OOM, panic, timeout, BE loss, or protocol ambiguity after dispatch | indeterminate | indeterminate | indeterminate | Three properties of the pinned ABI shape this table and are normative context, not implementation detail: - only the exact commit-conflict variant maps to the dedicated conflict code in the pinned C binding; other provider conflict variants share the aggregate internal code. A same-name race may therefore surface as a generic internal error, which classifies as indeterminate. Doris never parses retry hints or any message text to improve attribution; - dataset-not-found, generic not-found, and index-not-found share one code, and the drop path includes internal reads as well as the commit. A generic not-found received after the mutating invocation began therefore cannot prove a clean missing-target no-commit — for DROP as for every operation it classifies as indeterminate. A DROP whose authoritative preflight proved the name absent resolves as an `IF EXISTS` no-op **before** the mutating invocation; there is no post-invocation DROP no-op optimization in 4.2; - aggregate error classes lose information by design: some outcomes that did not commit will report indeterminate. This is a property of the pinned ABI and is accepted. Duplicate CREATE may return coarse `INDEX` or `INVALID_ARGUMENT`, not a stable already-exists/no-commit result. CREATE IF NOT EXISTS therefore resolves before dispatch; a later matching index is only corroboration and the outcome stays indeterminate. ### 6.6 Error surface Errors are typed, bounded, and sanitized. The four outcome classes of Section 1.1 are distinguishable by the client through distinct error codes/errClasses following Doris conventions; exact codes are implementation details. No secrets, raw provider payloads, unbounded text, or fabricated progress appear in any error. ## 7. Authorization, confidentiality, and cleanup - CREATE, REPLACE, and DROP require table `ALTER`. - SHOW INDEX and the physical inspection TVF require table `SHOW`. - Refresh actions run as a system action only after the initiating statement passed authorization, including the cleanup continuation that survives a client disconnect (Section 5.5). - There is no job lookup surface and no operator release path. - Secrets are excluded from durable state (none exists for mutations), child argv/environment, retry buffers, logs, core, SQL results, error messages, and test artifacts. Failed native work may leave unreferenced index files. Doris never guesses and deletes physical paths. Lance cleanup/VACUUM and historical-version retention own physical reclamation. ## 8. Distributed construction extension (future work, non-normative) Synchronous SQL does not preclude distributed index construction. The desired extension is: ```text pin snapshot -> partition fragments -> parallel BE builds of uncommitted segments -> validate/collect outputs -> one coordinated final commit -> refresh -> return ``` - build and commit responsibilities MUST remain separable in the synchronous implementation so this path can be added without redesign; - the baseline bindings (Section 0) expose both halves of a two-phase build: `lance_index_segment_builder_execute_uncommitted` returns protobuf-encoded segment metadata without touching the manifest, and `lance_dataset_commit_index_segments` commits pre-built segments as one logical index in a single manifest commit; the FE binding exposes `commitExistingIndexSegments`. Entry-point existence is not sufficient evidence: this path still requires Doris-side orchestration, transport of the uncommitted segment artifacts, verification of the segment contract, object-storage visibility, model compatibility, and exactly-one coordinated commit before implementation; - the existing segment-commit entry carries coverage-driven replacement semantics and no plain-CREATE `replace=false` contract. The initial version therefore keeps the single one-shot mutating call, and any two-phase capability is designed and verified separately — CREATE/IF semantics are not changed for extensibility; - Doris MUST NOT run several complete, independently committing `create_index` calls against the same logical index name; - the initial implementation uses one worker; multi-BE construction can follow without introducing durable jobs; persistent recovery remains a separate, justified follow-up requirement rather than a prerequisite for index lifecycle support. ## 9. Delivery and release evidence ### 9.1 Reviewable delivery slices 1. explicit revert/cleanup on `branch-4.1`: remove the durable job framework of #67235 and the job-specific parts of #67630 (job creation, JobId results, job inspection SQL, unresolved-job guards), preserving and adapting the privilege/schema/parameter/snapshot/IF validation, `SHOW INDEX`, and the physical TVF. Compatibility verification targets: the removed operation code keeps its tombstone and is never reused; the journal reader still decodes an old record; edit-log replay discards it; old image modules are skipped; and the JobId removal is checked against ordinary MySQL, forwarded, and Flight SQL result schemas. Existing metadata must not become unreadable; 2. synchronous execution path: master forwarding and the active-name entry; final in-entry preflight; dispatch, wait, classify, refresh, and return; the indeterminate outcome class and the committed, refresh incomplete class; budget and capacity accounting; 3. reworked isolated worker around the synchronous lifecycle — isolation, supervisor, budget enforcement, and domain-wide cleanup carry over; 4. fault, isolation, and query-consumption tests rewritten against Section 6; 5. BTREE/BITMAP lifecycle and user documentation. Mutation remains disabled until the evidence for its slice passes. ### 9.2 Gates - **G1 — SQL and metadata:** every CREATE/REPLACE/DROP/IF case; type/property rules at admission and exact-snapshot revalidation, verified against datasets that have round-tripped through a real manifest rather than only in-memory constructed schemas; DROP and DROP IF EXISTS success paths for composite, nested, and legacy indexes, including indexes with no readable logical description or schema-contract payload, and a case where the admitted version is still readable while storage-latest has advanced (the drop proceeds from the pinned snapshot and any race classifies per Section 6.5); one-snapshot SHOW/TVF behavior including normalization, system-entry filtering, bounds, and fail-closed malformed metadata; legacy name occupancy determined from the physical list, including the legacy per-name empty-result exception; `IF` no-op invalidation and refresh initiation including their failure and timeout paths — `CREATE IF NOT EXISTS` and `DROP IF EXISTS` no-op cases assert zero execu te sends, zero workers, the correct SQL class, name/target owner release, that a repaired refresh is retried only through a fresh authoritative preflight, and that a follower-initiated statement followed by a read on another FE observes the existing refresh-propagation semantics; case-only collisions; privileges. - **G2 — synchronous lifecycle correctness:** the classification table exercised over fault permutations (lost response, timeout, OOM, panic, signal, BE loss, malformed protocol, identity mismatch); no automatic retry after possible dispatch; refresh-failure reported distinctly, and the no-op refresh closure classified in the confirmed-failure family (never committed, never indeterminate); only identity-matched responses classify outcomes. Every fault case asserts three things — what the client observes, whether caches were refreshed per the responsibility table (Section 6.4), and when the executor, its capacity slot, and the active-name entry are released; asserting only a SQL error or an execution count is insufficient. Two mandatory window cases: **success received, then statement timeout** must report committed, refresh incomplete (never indeterminate); **timeout returned, then a new same-name admission** must observe busy until the execution domain terminates, then procee d from a fresh preflight. - **G3 — resource safety:** unavailable isolation rejects before invocation; FE and BE survive worker OOM, panics, malformed protocol, and signals; domain-wide reaping including descendants; capacity ownership holds under admission, budget expiry, reap, and BE-epoch loss; bounded metadata reads; credential rotation; secret non-disclosure; two-tier queueing with dequeue revalidation; delayed delivery, queue dwell, FE loss, cancel arriving before execute, and clock-skew boundaries — no expired mutation ever starts, with insufficient budget plus complete no-invocation evidence a confirmed failure and without it indeterminate; a lost-termination-report case — the BE confirms domain termination while the FE loses every report, after which network recovery follows the cleanup rules, consecutive different-name executions respect the retained-owner total, and same-name admission is never released by TTL or heartbeat guesses. - **G4 — end-to-end usability and documentation:** for each enabled index type, one focused fixed-snapshot end-to-end test creates the index through the isolated worker and proves the current query planning path consumes it; documentation covers SQL, synchronous semantics, the four outcome classes, inspection, worker settings, and Section 10's limitations. ### 9.3 Safety configuration invariants The 4.2 contract does not fix setting names. Mutation defaults disabled and cannot be enabled until the implementation has a verified hard worker memory/PID/runtime boundary, finite concurrency and queue bounds, a configured total time budget covering reads, queueing, execution, and refresh, and bounded metadata/input processing. Exact settings and reviewed defaults follow Doris conventions. ## 10. Accepted operational limitations - an indeterminate outcome affects resubmission decisions: there is no attribution, no automatic reconciliation, and no operator release path, and a blind resubmission may duplicate or race a mutation that committed without being reported; - same-name serialization is master-local: admission preflight plus the in-memory active-name entry. A busy entry temporarily blocks new same-name requests until the execution domain terminates — and, when the termination proof is lost, until the coordinator epoch is rebuilt. There is no serialization across FE failover or against external writers — an accepted window; - FE restart loses the in-memory control plane — name entries, target tokens, budgets, and FE-side accounting. In-flight clients observe a disconnect and follow the indeterminate rule; worker capacity on BEs is unaffected and drains by its own evidence; - classification loses information to aggregate provider codes: some outcomes that did not commit report indeterminate (distinct conflict and not-found variants share codes in the pinned ABI). Message text is never consulted to improve attribution; - cache freshness: committed indexes and authoritative no-ops trigger bounded refresh inside the statement, but a refresh failure leaves caches stale until an explicit refresh; authoritative SHOW/TVF reads remain available throughout; - one worker and one selected BE per mutation initially; no native progress, no resumable builds, cancellation is best-effort termination and never rollback, and there is no continuously appended-table freshness SLA; - query planning: branch-4.1's existing planner already generates splits for vector index segments and uncovered fragments; this issue adds no query architecture, and each enabled index type must prove the current path consumes it; - initial type, metric, nullability, schema, and property limits for CREATE and REPLACE remain strict; vector element nullability is not among them (it is unobservable at the schema level, Section 2.4). ## 11. Review request **Verified baseline evidence.** The four facts below were corrected against the pinned sources on 2026-10-09 and shape the contract above; please flag any disagreement: 1. the commit-resolved baseline: Doris `branch-4.1` `c8c0ddca4d`; FE `lance-core` `12.0.0`; BE `lance-c` commit `98468344` (version string `0.1.9`; vars.sh-pinned; Rust core `68c12dfd`) — package versions and release tags alone do not determine ABI behavior; 2. panic handling in the pinned C binding: panics are caught at the FFI boundary and reported as a typed panic error; unwinding-incompatible build settings are rejected at compile time; OOM, stack overflow, and double panics can still terminate a process — which is why isolation requirements are retained on resource grounds; 3. coordinated-commit availability: the FE binding exposes `commitExistingIndexSegments`, and the pinned C binding exposes both the uncommitted build half (`lance_index_segment_builder_execute_uncommitted`) and a C-side segment-commit entry (`lance_dataset_commit_index_segments`); 4. error mapping: only the exact commit-conflict variant reaches the dedicated conflict code; other conflict variants share the aggregate internal code, and dataset/generic/index not-found share one code — so post-invocation aggregate codes cannot prove non-commit. **The single open decision — client-disconnect semantics.** A plain disconnect after dispatch lets the dispatched work finish within its budget: no commit is guaranteed, no result packet can be delivered, and the client treats the outcome as indeterminate (Sections 1.1 and 5.5). Explicit KILL or statement timeout, by contrast, requests best-effort termination; a complete success already received before cancellation is still reported as committed, refresh incomplete rather than indeterminate (Section 6.3). This is the one place the design departs from Doris convention, which cancels queries when the client disconnects. The choice is a product trade-off, stated honestly: terminating on disconnect as well is coherent, would free the worker's CPU, memory, and slot sooner, and a disconnected client observes indeterminate under either policy — it cannot receive a result packet either way. Letting dispatched work finish is recommended because the work was explicitly requested, its co mpletion becomes authoritatively inspectable, and its continued resource use is already bounded by the statement budget. Confirmation or rejection of this single choice is requested. **Boundary confirmation.** Please confirm whether this document accurately reflects the final Doris 4.2 implementation boundary. A simple **Yes** would be helpful; if not, please identify any specific conflict with **Required in 4.2** (Section 1.2) or **Explicitly Deferred** (Section 1.3). Please review this comment-thread document as the complete proposal; no requirement should be inferred from v5/v5.1 or from the review history they capture. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
