u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5241658396
# Design Proposal v5 (Revised) — Lance Index Lifecycle for Doris 4.2 - Status: revised publication candidate - Target: Apache Doris `branch-4.1` / Doris 4.2 - Issue: [apache/doris#66497](https://github.com/apache/doris/issues/66497) - Source: full v5 proposal at commit `dec38e63f` - Supersedes: proposals v1 through v4.1; supersedes the source v5 after approval ## 0. Authority, evidence, and review request This is a standalone revision of v5. It preserves the lifecycle, failure, and acceptance contract of the source proposal while giving each requirement one normative definition. Appendices in this document are normative; implementation notes are explicitly marked non-normative. Statements have three meanings: - **Baseline fact**: verified in the target Doris commit or pinned dependency. - **4.2 decision**: required behavior and acceptance evidence. - **Future work**: behavior that 4.2 must not promise. The baseline is Doris commit [`e3289c1a5df7558cb8e63d80379d4edebf9c498c`](https://github.com/apache/doris/commit/e3289c1a5df7558cb8e63d80379d4edebf9c498c), with FE `lance-core` `9.1.0-beta.3`, Lance Namespace `0.7.7`, and BE `lance-c` `0.1.2` (Rust Lance `4.0.1`). No 4.2 guarantee depends on upgrading or extending those APIs. Approval is requested for these decisions: 1. common read-only inspection lands before mutation; 2. mutation is Directory-only and uses the pinned one-shot APIs; 3. logical metadata/count and physical UUID/version are separate surfaces; 4. native mutation runs in a hard-limited worker process, never in FE or BE; 5. dispatched work has no automatic retry or running cancellation; 6. outcome, refresh, reconciliation, disposition, and resource accounting are durable but independent; 7. an unknown job keeps its same-name fence until an audited force release; 8. mutation remains disabled until the release evidence in Section 10 passes. ## 1. Scope and release boundary ### 1.1 Goals Doris 4.2 provides: - authoritative Directory and REST `SHOW INDEX` support; - an exact logical-index count and bounded physical UUID/version inspection; - Directory CREATE, full same-name REPLACE, and DROP; - IVF_PQ vector and BTREE/BITMAP scalar indexes; - asynchronous durable jobs for every Directory mutation; - explicit naming, version, privilege, concurrency, failure, refresh, resource, and operator-resolution behavior; - cross-SDK and query-I/O evidence that the created indexes are readable and actually consumed. Lance manifests are authoritative for Directory metadata. The Namespace service is authoritative for REST metadata. Doris persists mutation requests and job control, not a second copy of the current external index definition. ### 1.2 Non-goals The following are not provided in 4.2: - incremental `BUILD INDEX`, distributed fragment builds, segment merge, or FE-side segment commit; - native progress or reliable cancellation after dispatch; - automatic retry of a dispatched mutation; - attribution from a matching metadata postcondition; - REST mutation under the generic Namespace contract; - IVF_FLAT, HNSW, FTS, composite indexes, nullable indexed columns, additional vector dtypes, 4-bit PQ, or user-supplied scalar tuning JSON; - fragment IDs, opaque index details, runtime hints, or unbounded arrays; - proof that two processes share the same local mount namespace. `BUILD INDEX` on a Lance table fails before job creation and names the incremental-build deferral. ## 2. User-visible contract ### 2.1 SQL ```sql CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (embedding) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "l2", "num_partitions" = "256", "num_sub_vectors" = "16" ); CREATE INDEX idx_btree ON lance_ctl.db.tbl (event_time) USING BTREE; CREATE INDEX idx_bitmap ON lance_ctl.db.tbl (category) USING BITMAP; CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (embedding) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "cosine", "num_partitions" = "256", "num_sub_vectors" = "16" ); SHOW INDEX FROM lance_ctl.db.tbl; SELECT COUNT(*) AS logical_index_count FROM lance_indexes("table" = "lance_ctl.db.tbl"); SELECT * FROM lance_index_entries("table" = "lance_ctl.db.tbl") WHERE index_name = "idx"; DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl; ``` `CREATE OR REPLACE` maps to `replace=true` and is a full rebuild. It is convergent by name but not idempotent: another invocation may create a different UUID and dataset version. `IF NOT EXISTS` and `OR REPLACE` are mutually exclusive. `USING` is mandatory. Unknown types, properties, or values fail before job admission. `USING ANN` only reuses the neutral DDL category; it neither adds nor implies Doris internal ANN query syntax for Lance. Lance queries continue to use `vector_search()`. CREATE, REPLACE, and DROP return when the job and same-name fence are durable, not when native work finishes. The OK message includes the job ID. A client that loses the response must inspect jobs rather than blindly resubmit. Top-level CREATE/REPLACE/DROP is the only 4.2 mutation syntax. `ALTER TABLE ... ADD/DROP INDEX` remains unsupported for Lance and unchanged for internal tables. ### 2.2 Job control ```sql SHOW LANCE INDEX JOBS [FROM lance_ctl.db] [WHERE TableName = "tbl" AND State = "OUTCOME_UNKNOWN"]; SHOW LANCE INDEX JOB <job_id>; CANCEL LANCE INDEX JOB <job_id>; RESOLVE LANCE INDEX JOB <job_id> AS ACKNOWLEDGED COMMENT '<non-empty operational note>'; RESOLVE LANCE INDEX JOB <job_id> AS FORCE_RELEASE COMMENT '<non-empty risk acceptance note>'; RESOLVE LANCE INDEX JOB <job_id> AS FORCE_RELEASE WITHOUT REFRESH COMMENT '<ADMIN emergency note>'; ``` `CANCEL` is legal only while execution is `PENDING`. If dispatch has already won the expected-state transition, the command returns a typed not-cancellable-after-dispatch error. Running cancellation is not offered even though the isolated child can be killed: a kill at the manifest commit boundary has outcome `UNKNOWN` and cannot satisfy SQL cancellation semantics. There is no state in which a job is reported `CANCELLED` while an external mutation may still commit. This is separate from internal `SHOW/CANCEL BUILD INDEX`: that path is tied to `OlapTable`, internal proc nodes, and replay-resend semantics that are unsafe for an external one-shot commit. Job list columns are: `JobId, CatalogName, DatabaseName, TableName, IndexName, Operation, CreateTime, StartTime, FinishTime, State, Outcome, RefreshState, ReconcileState, Disposition, Executor, Message, LastResolutionAction, LastResolvedBy, LastResolvedAt`. The detail form also returns bounded immutable resolution events: action, actor, request/completion time, result, note, skipped-refresh flag, and warning. Notes are trimmed, sanitized, and limited to 1,024 UTF-8 bytes. There is no percentage progress because the pinned mutation API exposes none. ### 2.3 Type and property matrix | Type | Column contract | Properties and constraints | |---|---|---| | `IVF_PQ` | one non-null fixed-size-list of non-null `FLOAT16` or `FLOAT32` | `index_type=IVF_PQ`; `metric=l2|cosine|dot`; required positive `num_partitions` and `num_sub_vectors`; subvectors divide dimension; partitions do not exceed snapshot rows; `num_bits=8` is fixed | | `BTREE` | one non-null scalar from integral, floating, decimal, string, date, or timestamp predicate-pushdown types | no user build properties; C `params_json=NULL` | | `BITMAP` | one non-null boolean, integral, string, or date column | no user build properties; C `params_json=NULL` | Hamming, nullable or nested fields, FLOAT64/UINT8/INT8 vector elements, composite indexes, and arbitrary JSON are rejected. Administrator safety caps for rows, fragments, dimension, partitions, and subvectors apply before FFI. ## 3. Pinned provider contract ### 3.1 Directory The mutation path uses only synchronous `lance-c` `0.1.2` one-shot create-vector, create-scalar, and drop calls. They expose no cancel token, progress callback, task handle, operation ID, status query, idempotency key, or preassigned UUID. A successful matching response proves commit; loss of that response does not. ```c int32_t lance_dataset_create_vector_index( LanceDataset*, const char* column, const char* name, const LanceVectorIndexParams*, bool replace); int32_t lance_dataset_create_scalar_index( LanceDataset*, const char* column, const char* name, LanceScalarIndexType, const char* params_json, bool replace); int32_t lance_dataset_drop_index(LanceDataset*, const char* name); uint64_t lance_dataset_index_count(const LanceDataset*); const char* lance_dataset_index_list_json(const LanceDataset*); ``` The BE wrapper supplies an explicit deterministic name, validates every enum and property before FFI, reads the thread-local error code before the consuming message, never parses message text, checks the error code when index count is zero, frees returned JSON with `lance_free_string`, and retains only a typed code plus bounded sanitized message. FE authoritative reads use one Java-SDK snapshot: - `Dataset.describeIndices()` without criteria for logical descriptions, avoiding legacy missing-details false negatives; - that snapshot's schema to resolve field IDs; - `Dataset.getIndexes()` for physical UUIDs and dataset versions; - `Dataset.countRows()` when row context is required. `Dataset.getIndexStatistics()` is forbidden in SHOW, IF, and reconciliation because the pinned implementation can migrate legacy metadata and write a new manifest. `detailsJson` is parsed through a fixed allowlist and never returned verbatim. `target_partition_size` is not relabeled as `num_partitions`. ### 3.2 REST Namespace `0.7.7` has create/list/stats/drop/transaction methods but lacks a safe generic mutation contract: replace and IVF training fields cannot be expressed; transaction IDs are optional; index status is provider-defined; transaction responses do not bind table/index/operation/UUID; cancel does not guarantee commit prevention; and list/stats postconditions provide no writer attribution. Therefore 4.2 has one code-defined REST profile, `generic-read-only-v1`: - bounded list/stats inspection is allowed; - create, replace, build, drop, and cancel fail before job/fence creation; - no URI inference, mutation probe, user status map, or direct-Dataset fallback is allowed. A provider-specific REST mutation profile is future work, not framework shipped by this issue. ## 4. Metadata contract ### 4.1 Logical surfaces `SHOW INDEX` retains the MySQL-compatible 13-column schema and returns one row per logical index/column. For Lance, `Key_name`, `Column_name`, and `Index_type` come from authoritative logical metadata; `Index_type` is the physical provider type (for example `IVF_PQ/BTREE/BITMAP`), not SQL category `ANN`. A bounded externally created type may be inspected without becoming creatable by Doris. `Properties` is deterministic, valid, allowlisted JSON rather than request history or opaque provider data. `lance_indexes("table"="ctl.db.tbl")` returns one row per exact-case logical name: `CatalogName, DatabaseName, TableName, IndexName, Columns, IndexType, RowsIndexed, PhysicalEntryCount, Properties`. `COUNT(*)` on that TVF is the exact user-visible logical count. It excludes system indexes, is independent of physical-entry and column count, and counts case-only external names separately. Directory builds logical and physical views from `describeIndices()` and `getIndexes()` on the same snapshot. REST builds the logical view from complete pagination and at most one stats request per exact name. REST logical inspection allows at most 256 names, four concurrent stats requests, and one statement deadline; a failed stats call fails the statement rather than returning partial types. Logical inspection fails with `LANCE_INDEX_METADATA_INCONSISTENT` if a physical entry is orphaned, a logical description lacks a required entry, same-name fields/types disagree, a UUID belongs to two names, or a field cannot be resolved. It never silently merges or drops corrupt metadata. ### 4.2 Physical surface `lance_index_entries` returns one row per physical entry: `CatalogName, DatabaseName, TableName, IndexName, IndexUuid, Columns, IndexType, DatasetVersion, ProviderStatus, MetadataConsistency`. | Provider | Name/UUID/columns | Type | Version | Status | Consistency | |---|---|---|---|---|---| | Directory | `getIndexes()` | `Index.indexType` | `Index.datasetVersion` | `NULL` | same-snapshot logical join | | REST | paginated list | `NULL`; the physical TVF issues no per-entry stats fan-out | `NULL` | bounded raw provider string, display only | list-level exact-name/UUID checks, otherwise `UNKNOWN` | Consistency is `CONSISTENT`, `ORPHAN`, `MISMATCH`, or `UNKNOWN`. The physical TVF may expose diagnostic rows when logical inspection fails, but never exposes fragments, opaque details, coverage arrays, or transaction properties. Both TVFs require a table argument and table `SHOW`. Results are all-or-error, never silently truncated. Normative field allowlists and bounds are in Appendix A. ## 5. Naming, snapshots, and command routing Doris preserves Lance display case but matches user input case-insensitively. New case-only duplicates are rejected; a unique match is sent with its exact stored name; an externally created case-only collision is displayed but blocks mutation as ambiguous. The fence key is: ```text (catalog incarnation, table incarnation, normalization version, persisted normalized logical-name bytes) ``` Normalization v1 is UTF-8 of Java `toLowerCase(Locale.ROOT)`. It is a versioned Doris comparison rule, not full Unicode case folding. Exact display name and normalized bytes are both persisted; replay never recomputes an old key. The serialization guarantee is scoped to one catalog/table incarnation. URI aliases are external writers, and credential-bearing URLs are never identity. Historical snapshots are read-only for mutation. Admission records latest version and schema fingerprint. The child reopens that exact version and revalidates schema and resource inputs before setting `invoked=true`. A mismatch is `STALE_ADMISSION/NOT_COMMITTED`, requires refresh, and is manually re-issued; the worker never switches silently to latest. Lance may permit later append, which can leave uncovered fragments. `IF NOT EXISTS` compares only authoritative readable fields: name, columns, physical type, and allowlisted requested fields. Unreadable historical build parameters are not reconstructed from a Doris job. Target-aware provider dispatch occurs before translation to the internal `catalog.Index`. Internal behavior remains unchanged; Lance definitions do not extend the internal physical `IndexType` enum. All mutations and job commands forward-with-sync to the current master. SHOW/TVFs bypass cached index definitions and open one authoritative selected/latest snapshot. Job SHOW is leader-linearizable. ## 6. Directory execution boundary ```text durable FE job + fence -> FE catalog/cluster admission -> selected BE atomic reservation -> durable RUNNING(BE, attempt, generation, leader epoch, deadline) -> dedicated BE supervisor -> isolated worker process -> pinned one-shot mutation -> typed result classification -> authoritative FE read and replayable refresh ``` FE metadata and DDL locks are held only for admission, guard validation, and durable transitions. They are released before BE dispatch, native execution, authoritative reread, and refresh retry. The pinned Rust runtime creates threads outside Doris task `MemTracker` control and is built with `panic=abort`. Production mutation therefore never runs in FE or the BE process. The selected BE launches a same-build worker inside a verified OS hard memory/process boundary before the dynamic loader or Lance/Rust static initialization can allocate materially; a deployment that cannot provide that boundary rejects mutation. There is no production in-process or soft-RSS fallback. The worker has hard memory and PID limits, fd/core restrictions, a wall-clock deadline, parent-death behavior, closed inherited fds, an allowlisted environment, dump disabled, and a protocol/build/ABI handshake. An operator- delegated cgroup v2 parent or pre-created pool is required on Linux. The launcher may use `clone3(CLONE_INTO_CGROUP)` or an equivalent stop/move/verify/exec barrier; the safety boundary, not one syscall, is normative. Current catalog credentials are resolved at dispatch. Doris preserves temporary credentials supplied by the provider but does not mint or narrow them. Static credentials are allowed and must be storage-scoped by the operator. Secrets are excluded from edit log, argv, environment, logs, core, and retry buffers; they travel through sensitive RPC plus an anonymous child pipe and are zeroized after use. Each BE has a bounded dedicated supervisor, default concurrency one, no queue by default (at most one explicitly configured waiting item), atomic attempt de-duplication, and no ordinary RPC-thread FFI. Worker responses include attempt identity, `invoked`, native return, saved typed error, and bounded message. BE restart/shutdown kills and reaps children or relies on parent-death behavior; cgroup cleanup is idempotent. A pre-invocation busy rejection releases that reservation and leaves `PENDING/NONE`; another BE may be tried. Scheduling-policy exhaustion makes one terminal `NOT_COMMITTED/NOT_REQUIRED` transition. Once RUNNING is durable, no mutation is automatically redispatched. `invoked` is trusted only in a complete, identity-matched worker result. The supervisor may prove no invocation before request delivery or worker launch, during a failed handshake, or from a complete structured `invoked=false` result. After request acceptance, EOF, signal, timeout, BE loss, or any missing/partial result is `UNKNOWN`; absence of an `invoked=true` marker is never proof of non-invocation. v5 defines no streaming or durable invocation marker. Wall-clock expiry bounds FE catalog/cluster accounting leases but does not prove worker termination. BE-attempt accounting follows the actual child until a matching kill-and-reap acknowledgement or BE process-epoch proof. EOF, signal, OOM, protocol corruption, or identity mismatch is `UNKNOWN` after invocation. Local/`file://` mutation has a separate disabled-by-default operator assertion, requires exactly one FE and BE, normalized absolute paths, pre/post version and identity checks, and rejection on topology change before dispatch. These checks do not prove a shared mount namespace. Object storage is the production mode. Restricted SQL users remain untrusted and are contained by table RBAC; FE-to-BE peers and administrator-configured storage are operator-trusted under the Doris threat model. The child boundary contains crashes and resource use; it is not a new tenant-security sandbox or a claim of authenticated/confidential internal RPC. ## 7. Durable lifecycle contract ### 7.1 State dimensions and legal combinations ```text execution: PENDING | CANCELLING | RUNNING | TERMINAL outcome: NONE | COMMITTED | NOT_COMMITTED | NOOP | CANCELLED_BEFORE_DISPATCH | UNKNOWN refresh: NOT_REQUIRED | REQUIRED | RUNNING | DONE | RETRY_PENDING | SKIPPED_BY_FORCE reconciliation:NONE | WATCHING | RETRY_PENDING | FINAL_READ_PENDING | STOPPED_TERMINATION_PROVEN | STOPPED_BY_FORCE disposition: NONE | ACK_REFRESH_PENDING | ACKNOWLEDGED | FORCE_REFRESH_PENDING | FORCE_RELEASED ``` Legal-combination rules: - nonterminal execution has outcome `NONE`; - `WATCHING` and `FINAL_READ_PENDING` apply only to `UNKNOWN`; - ACK never releases the same-name fence; - a known outcome retains the fence while required refresh is unfinished; - `SKIPPED_BY_FORCE` requires `FORCE_RELEASED` and its ADMIN audit completion; - `FORCE_RELEASED` never changes the original outcome. The SHOW `State` column is derived rather than persisted as another authority. Values include `PENDING`, `RUNNING`, `REFRESHING`, `FINISHED`, `FAILED`, `FAILED_REFRESH_PENDING`, `COMMITTED_REFRESH_PENDING`, `COMMITTED_REFRESH_SKIPPED`, `FAILED_REFRESH_SKIPPED`, `CANCELLED`, and `OUTCOME_UNKNOWN`. A skipped refresh is never displayed as coherent `FINISHED`. ### 7.2 Correctness invariants **I1. Durable send boundary.** `RUNNING` with BE, attempt, generation, leader epoch, and deadline is durable before execute send; leadership and revision are checked again immediately before network I/O. **I2. No redispatch.** A dispatched one-shot mutation is never automatically retried without a provider idempotency contract. **I3. Attribution is not observation.** Name presence/absence, matching fields, or version increase cannot turn a lost-response Directory job into a known outcome. **I4. Orthogonality.** Outcome, refresh, reconciliation, disposition, and execution accounting have independent persisted lifetimes. **I5. Correctness fence.** If an old executor may still commit, the same-name fence remains until a durable privileged FORCE accepts the risk. ACK, retention, archival, deadline, termination proof, and slot release do not release it. **I6. Slots are not fences.** FE accounting leases may expire; BE ownership needs matching termination proof. Neither action changes attribution. **I7. Replayable effects.** Fence/slot/watcher/disposition changes derive from durable transitions, matching attempt proof, or persisted leases. Refresh is persisted REQUIRED before its side effect and DONE afterwards. **I8. Durable tombstone.** Archival retains everything needed to rebuild an unknown fence, guard ownership, resolution audit, privilege fallback, watcher, and SHOW/RESOLVE behavior. ### 7.3 Canonical transitions | ID | Trigger and CAS source | Persisted result | Refresh/reconciliation | Resource and fence effect | |---|---|---|---|---| | T1 | validated admission under target guard | `PENDING/NONE` | `NOT_REQUIRED/NONE` | create job+fence atomically; no running slot | | T2 | CANCEL wins `PENDING` | `CANCELLING`, then `TERMINAL/CANCELLED_BEFORE_DISPATCH` | none | prove no send; release reservation/fence | | T3 | dispatcher wins `PENDING` | `RUNNING/NONE` with attempt identity | none | acquire running accounting before send | | T4 | matching attempt supplies explicit proof that it was never invoked | `TERMINAL/NOT_COMMITTED` | refresh if an authoritative admission, pre-dispatch, or worker-revalidation observation detected dataset-version/fingerprint advancement or relevant metadata change | release accounting; release fence after refresh obligation | | T5 | matching native success | `TERMINAL/COMMITTED` | `REQUIRED` | release accounting; hold fence until DONE | | T6 | pinned no-commit, no external change | `TERMINAL/NOT_COMMITTED` | `NOT_REQUIRED` | release accounting and fence | | T7 | pinned no-commit plus conflict/existence change | `TERMINAL/NOT_COMMITTED` or IF `NOOP` | `REQUIRED` | release accounting; hold fence until DONE | | T8 | lost/invalid result or aggregate post-invocation error | `TERMINAL/UNKNOWN` | `WATCHING`; refresh on observed change | release proven-dead accounting only; retain fence | | T9 | watcher observes version/index change | outcome unchanged | atomically set `REQUIRED`; continue watching | retain fence | | T10 | matching termination proof | outcome `UNKNOWN` unchanged | `FINAL_READ_PENDING`, then `STOPPED_TERMINATION_PROVEN` after successful read | release BE ownership; retain fence | | T11 | refresh worker | outcome unchanged | `REQUIRED/RETRY_PENDING -> RUNNING -> DONE` | release eligible known-outcome fence only after DONE | | T12 | ACK request/complete | outcome `UNKNOWN`; `ACK_REFRESH_PENDING -> ACKNOWLEDGED` | authoritative refresh to DONE | append linked audit events; retain fence | | T13 | ordinary FORCE request/complete | outcome unchanged; `FORCE_REFRESH_PENDING -> FORCE_RELEASED` | authoritative refresh to DONE; stop watcher | append events/warning; release remaining leases/fence | | T14 | ADMIN FORCE WITHOUT REFRESH | preserve terminal outcome; `FORCE_RELEASED` | `SKIPPED_BY_FORCE/STOPPED_BY_FORCE` | supersede pending resolution, append emergency events/warning, release leases/fence in one CAS | Every transition verifies current master/leader epoch, expected revision, expected execution/disposition, and, where applicable, attempt, dispatch generation, and refresh generation. It appends a replayable `revision+1` record. A demoted leader cannot mutate local durable state; stale callbacks are rejected. ### 7.4 Provider result classification The adapter never classifies message text. It sets its local `invoked=true` flag immediately before entering `lance_dataset_*` and returns that value only in a complete terminal result; the flag is not a streaming or durable phase marker. | Saved result | CREATE | REPLACE | DROP | Refresh obligation | |---|---|---|---|---| | rejection before FFI | `NOT_COMMITTED` if a job exists | same | same | if an authoritative admission, pre-dispatch, or worker-revalidation observation detected advancement or relevant metadata change | | success | `COMMITTED` | `COMMITTED` | `COMMITTED` | required | | `CommitConflict` after invocation | `NOT_COMMITTED` | `NOT_COMMITTED` | `NOT_COMMITTED` | required | | `NotFound` after invocation | `UNKNOWN` | `UNKNOWN` | `NOT_COMMITTED`, or `NOOP` for IF EXISTS | required for DROP; reconcile otherwise | | `InvalidArgument` or `NotSupported` after invocation | `UNKNOWN` | `UNKNOWN` | `UNKNOWN` | reconcile | | `Index`, `IO`, or `Internal` after invocation | `UNKNOWN` | `UNKNOWN` | `UNKNOWN` | reconcile | | no matching response, signal, OOM, BE loss, or protocol/transport ambiguity | `UNKNOWN` | `UNKNOWN` | `UNKNOWN` | reconcile | Authoritative preflight is an optimization and user-semantic check, not a lock against external writers. Every post-preflight race is still classified by the provider-result table. Duplicate CREATE may return `Index` or `InvalidArgument`; neither is a stable AlreadyExists/no-commit code. After invocation, plain CREATE and IF NOT EXISTS therefore remain `UNKNOWN` even when metadata later matches. Before dispatch, matching IF NOT EXISTS is a durable `NOOP`. DROP IF EXISTS preflight absence or typed DROP NotFound is `NOOP` plus refresh when a race/advancement was observed. ## 8. Unknown outcome, refresh, and operator resolution ### 8.1 Reconciliation and refresh An unknown Directory job owns a durable watcher. A master-only daemon reopens latest with Java SDK on persisted backoff (initial 30 seconds, capped at 30 minutes), records observed version/fingerprint, and schedules refresh whenever relevant state changes. Read failure is retryable; it changes neither outcome nor fence. Metadata match remains corroboration only. Deadline expiry alone does not stop the watcher. A matching reap or BE-epoch proof moves it to `FINAL_READ_PENDING`; after one successful final authoritative read it becomes `STOPPED_TERMINATION_PROVEN`. The outcome and fence remain UNKNOWN/held until FORCE. External-table refresh uses the existing idempotent cache invalidation and `OP_REFRESH_EXTERNAL_TABLE` replay path. The job records `REQUIRED` before refresh, then records `DONE` after the refresh edit log. Failover may repeat refresh but never mutation. FORCE_RELEASE stops the watcher; a later old commit is not guaranteed to be detected automatically and requires normal cache expiry, an authoritative metadata command, or explicit `REFRESH TABLE`. ### 8.2 Resolution | Action | Allowed source | Privilege | Durable protocol | Result | |---|---|---|---|---| | ACK | outcome `UNKNOWN`, disposition `NONE`; completed ACK retries return its existing events | target table `ALTER` | append immutable request; `ACK_REFRESH_PENDING`; refresh; append linked completion | `ACKNOWLEDGED`, outcome unchanged, fence held | | ordinary FORCE | outcome `UNKNOWN`, disposition `NONE` or `ACKNOWLEDGED`; termination proof is irrelevant | target `ALTER`, or `ADMIN` if target no longer resolves | append request; `FORCE_REFRESH_PENDING`; refresh; final CAS and completion | `FORCE_RELEASED`, outcome unchanged, fence released with warning | | FORCE WITHOUT REFRESH | any terminal refresh failure still owning the fence; `UNKNOWN` from `NONE`, `ACK_REFRESH_PENDING`, `ACKNOWLEDGED`, or `FORCE_REFRESH_PENDING` | `ADMIN` | one CAS increments refresh generation, completes a prior pending event as `SUPERSEDED_BY_EMERGENCY_FORCE`, then appends emergency request/completion | `SKIPPED_BY_FORCE/STOPPED_BY_FORCE/FORCE_RELEASED`; cache coherence waived | `RESOLVE` waits for refresh and final CAS. If refresh becomes `RETRY_PENDING`, it returns a typed incomplete-resolution error with job/resolution IDs; replay continues the same request. Retrying identical `(action, actor, normalized note)` joins the pending request. Different actions race by revision. Completed events are immutable and remain visible after archival. Older refresh-generation callbacks cannot restore refresh or guards. Every FORCE emits a SQL warning, audit log entry, and job-detail warning that an old executor may commit after release and overwrite, remove, or reintroduce the same index. `WITHOUT REFRESH` additionally warns that caches may remain stale. An unresolved or merely `ACKNOWLEDGED` `UNKNOWN` job is excluded from automatic terminal deletion. Retention may compact it only into the complete Appendix B tombstone, which remains SHOW/RESOLVE-addressable. Time alone proves neither worker termination, mutation attribution, nor correctness-fence release. ## 9. Authorization, concurrency, and cleanup - CREATE/REPLACE/DROP/CANCEL/ACK require table `ALTER`; ordinary FORCE uses table `ALTER` when the same incarnation resolves; missing-target and WITHOUT REFRESH force require global `ADMIN`. - `SHOW INDEX` and both metadata TVFs require table `SHOW`. Job list/detail uses table `SHOW` when the persisted target incarnation resolves. If it no longer resolves, the orphan job is visible only to global `ADMIN`; other listings omit it, and direct lookup returns the same non-disclosing result as a missing or unauthorized job. No denied path exposes names, paths, messages, executor identity, audit events, or counts. - A mutating job command first loads the persisted job by ID without returning any field, then authorizes against that persisted target. Only the documented ADMIN emergency path may proceed when the same target incarnation no longer resolves. - background refresh runs as the system job only after the initiating command's privilege check. Job admission, target-changing Doris DDL, and final FORCE use one lifecycle- keyed target guard independent of the table object, followed by object DDL lock when the incarnation resolves, target-guard manager lock, and job-manager lock. Admission appends job+fence before unlock. DDL appends only after proving no guard. FORCE appends release before DDL may proceed. Replay reconstructs guards before target-changing DDL, so exactly one edit-log order wins. A recreated object has a new incarnation and cannot satisfy an old job. `target-changing DDL` means any Doris-side operation that changes the catalog or table incarnation, provider/dataset locator, selected-version semantics, or the full schema fingerprint persisted at admission, including changes to unindexed columns. Cache-only refresh, data append, and metadata changes that preserve identity, locator, version selection, and the full schema fingerprint are not target-changing DDL. Doris serializes only its own jobs within one catalog/table incarnation and normalized name. External writers and URI aliases follow Lance conflict rules. Concurrent append may leave uncovered fragments; query correctness must combine indexed and unindexed data. Table/catalog DROP, RENAME, or target-changing schema DDL is rejected while an active/unknown table guard exists. Failed work may leave unreferenced files. Doris never deletes guessed paths; Lance cleanup/VACUUM and historical-version retention own physical reclamation. ## 10. Delivery, acceptance, and limitations ### 10.1 Phased delivery Following the maintainer preference, delivery is independently reviewable: 1. common authoritative Directory `SHOW INDEX`, preserving internal behavior; 2. logical count, physical UUID/version TVF, consistency handling, REST read-only; 3. mutation feasibility gate: both Java-produced -> Rust mutation -> Java reopen and Rust-produced -> Java reopen fixtures; 4. disabled FE mutation control: neutral routing, durable guards/jobs, job SQL, fake-provider fault tests; 5. disabled isolated worker plus IVF_PQ object-storage tracer bullet; 6. BTREE/BITMAP lifecycle and query-I/O proof; 7. local test mode, full user documentation, and enablement review. Read-only phases can release independently. Mutation remains disabled until phases 3–7 and all gates below pass together. ### 10.2 Release evidence | ID | Required evidence | |---|---| | G1 Cross-SDK | Record producer, manifest/storage format, and pinned versions. Java-produced latest -> Rust create/replace/drop -> Java authoritative reopen and reverse Rust-produced -> Java reopen cover IVF_PQ/BTREE/BITMAP. Unsupported formats fail pre-invocation as `UNSUPPORTED_LANCE_FORMAT`. | | G2 Query consumption | For L2, cosine, and dot, `vector_search(..., "use_index"="true")` plus scalar predicates run on a fresh cache while a query-phase recorder observes UUID-specific `_indices/<uuid>/...` lookup/data reads, not LIST/HEAD/prefetch. Vector `use_index=false` and scalar unusable-predicate plus dropped/no-index fixtures are negative controls. Results and uncovered-fragment reads are verified. | | G3 Isolation | Worker OOM, `panic=abort`, signal, deadline, and protocol failures leave BE alive; hard-boundary absence rejects pre-invocation; secrets are absent from edit log, image, argv, env, logs, core, SHOW, and retry buffers. | | G4 Lifecycle | Every T1–T14 transition and C1–C11 crash cut replays to the specified outcome, refresh, watcher, disposition, accounting, and fence. Unknown names cannot be reissued before FORCE. | | G5 Metadata/RBAC | Name/count/UUID/columns/type/version provenance, consistency errors, bounds, case collisions, SHOW/ALTER/ADMIN behavior, and no-leak denial are verified. | | G6 REST | All REST mutations fail before job creation; bounded list/stats and NULL/provenance behavior pass; no mutation-profile configuration exists. | | G7 Documentation | SQL, job lifecycle, resolve warnings, metadata/count semantics, privileges, settings, deployment prerequisites, failures, refresh, limitations, and troubleshooting are published. | The first mutation PR must also fix and document administrator settings for the feature gate, delegated cgroup parent, worker memory/runtime/PID limits, row/ fragment/vector/partition caps, and BE/catalog/cluster concurrency. Required zero/empty safety settings keep mutation disabled. | Setting | Default | Required effect | |---|---:|---| | `enable_lance_index_mutation` | `false` | false rejects before job creation | | `lance_index_worker_cgroup_parent` | empty | delegated parent/pool must pass BE startup capability verification | | `lance_index_worker_memory_bytes` | `0` | positive; applied as the hard memory limit | | `lance_index_worker_max_runtime_seconds` | `0` | positive; child deadline and upper bound for FE accounting leases, not termination proof | | `lance_index_worker_pids_max` | `0` | positive and sufficient for the pinned runtime | | `lance_index_max_rows` / `lance_index_max_fragments` | `0` | positive; checked by FE and exact worker snapshot | | `lance_index_max_vector_dimension` / `max_partitions` / `max_sub_vectors` | `0` | positive; checked before FFI | | BE/catalog/cluster concurrency | `1` | positive and bounded; BE remains final admission authority | ### 10.3 Accepted limitations - a dispatched Directory mutation may remain permanently `UNKNOWN`; - reconciliation cannot attribute a lost-response commit; - an unknown name remains fenced until privileged FORCE accepts late-commit risk; - CREATE/REPLACE is one whole-snapshot worker build without native progress; - running cancellation and incremental coverage repair are unavailable; - concurrent append may leave uncovered fragments; - generic REST mutation is unavailable; - local mutation relies on an operator assertion Doris cannot prove completely; - initial type, metric, nullability, and property limits are intentionally strict. Future work includes fragment builds, progress/cooperative cancel, durable queryable BE task identity, incremental maintenance, more index types, nullable/composite fields, and richer bounded metadata. Any future REST mutation profile must be a code-defined `(profile_id, contract_version)`, never URI-inferred or user-defined. For each operation it must define sync/async completion, mandatory transaction identity and retention, exact status mapping, operation binding/provenance, postconditions, typed no-commit failures, commit-preventing cancellation, idempotency, polling/backoff/deadline, and replay compatibility. ## Appendix A. Metadata allowlists and bounds (normative) Directory logical properties may contain only `metric_type`, `target_partition_size`, `compression_type`, `num_bits`, `num_sub_vectors`, `hnsw_max_connections`, `hnsw_construction_ef`, and `hnsw_max_level` when the pinned SDK supplies them. REST logical properties may contain only `distance_type`, `num_indexed_rows`, `num_unindexed_rows`, and `num_indices`. Provider status is displayed only in the physical REST surface and is never interpreted. `Properties` is deterministic valid JSON of at most 400 UTF-8 bytes. External strings must be valid UTF-8 and at most 1,024 bytes; each entry has at most 64 columns and 16 KiB total column-name bytes. REST requests use pages of at most 1,000, at most 100 pages and 10,000 physical entries; repeated page tokens are protocol errors. Raw maps/JSON/arrays are structurally bounded before formatting. Any overflow or malformed value fails closed; no result is truncated. ## Appendix B. Durable record and replay data (normative) The durable job contains: - job ID, monotonic revision, creator, and timestamps; - target catalog/database/table IDs, names and incarnations, stable locator, exact name, normalization version, and normalized fence key; - persisted provider `DIRECTORY` for every 4.2 mutation job; - operation, IF mode, columns, SQL category, physical type, allowlisted properties, exact starting version, and schema fingerprint; - execution state, selected BE and process epoch, reservation/attempt, generation, dispatch leader epoch, deadline, invoked/termination proof, typed error, and bounded sanitized message; - outcome/completion mode and last observed version/fingerprint; - refresh state/generation, retry/backoff/error; - reconciliation state and last/next observation; - disposition and immutable linked resolution request/completion events; - FE execution leases, BE-attempt ownership, same-name fence, and archive time. Credentials, vended URLs, headers, raw provider responses, opaque details, and unbounded text are forbidden. Full snapshots or deltas are acceptable only when every transition is edit-log ordered and replay produces the same revision. The tombstone must retain target identity/incarnation, locator, normalization key/version, provider, all state dimensions, BE attempt/deadline/ownership, leases/fence, resolution events, and fields required by SHOW, privilege fallback, watching, refresh, and FORCE. Archived jobs remain SHOW/RESOLVE addressable. ## Appendix C. Critical crash cuts (normative) | ID | Persistence/failure cut | Required replay | |---|---|---| | C1 | job/fence durable, client OK lost | replay PENDING; schedule once; client finds job through SHOW | | C2 | admission races target DDL, or CANCEL races reservation/dispatch | shared guard and expected-state CAS select one durable side; losing path sends nothing and releases reservation | | C3 | `CANCELLING` durable before terminal cancel | complete `CANCELLED_BEFORE_DISPATCH`; prove no execute send | | C4 | RUNNING durable before/during execute send, response lost, or silent worker loss after request acceptance even before FFI | `UNKNOWN`; lack of an invocation marker proves nothing; retain watcher/fence and never redispatch | | C5 | old leader callback/send races demotion | epoch/revision rejects mutation; if send may have executed, current leader keeps UNKNOWN/fence | | C6 | worker commits then response is lost | UNKNOWN; matching metadata is corroboration only | | C7 | hard deadline expires without reap proof | expire only persisted FE leases; retain BE ownership, watcher, fence, and UNKNOWN | | C8 | matching reap/old-BE termination proof arrives | release attempt ownership; final authoritative read to `STOPPED_TERMINATION_PROVEN`; retain UNKNOWN/fence | | C9 | outcome+REQUIRED durable before refresh, or local refresh before edit-log/DONE | retry refresh only; idempotently reach DONE; never replay mutation | | C10 | ACK/FORCE pending, refresh succeeds, final disposition not durable | replay the same refresh/generation and final CAS; retain guards until final record | | C11 | emergency/final FORCE or archive is durable before memory cleanup; old executor later commits | replay released/retained guards from record; old outcome stays UNKNOWN; late commit after FORCE is accepted risk and needs ordinary/explicit refresh | ## Appendix D. Verification traceability (normative) The implementation must include: - FE parser/to-SQL, routing, unchanged internal behavior, type/property/snapshot, `USING ANN` category/query separation, non-ASCII normalization/incarnation replay, typed post-dispatch CANCEL error, no `CANCELLED` state while commit is possible, orphan-job ADMIN visibility/no-leak, post-preflight external-writer races, unknown-job no-deletion/tombstone replay, transition/CAS/replay, metadata bounds, REST rejection, resolution/audit, and DDL-race unit tests covering every target-changing category and each documented exclusion; - BE ABI/error-order/freeing, supervisor/admission, worker handshake/secrets, hard-boundary/OOM/abort/reaping, pre/post-invocation failpoint, and typed result unit tests; - cross-SDK and object-storage integration for all three types, replace/drop, append/uncovered fragments, Java/BE metadata agreement, and UUID I/O proof; - regression coverage for SQL, logical count with multiple physical entries or columns, case-only collisions, metadata inconsistency, IF behavior, historical mutation rejection, privileges, and deterministic result order; - Doris-owned failpoints returning a complete structured `invoked=false` result before FFI, plus a silent kill at the same point that must remain `UNKNOWN`; additional failpoints after C return and before response; object-store proxy loss after manifest PUT acceptance but before response; and arbitrary post-invocation SIGKILL mapped to UNKNOWN without claiming an exact Lance-internal commit cut; - fault injection for every C1–C11 cut, FE leader change, BE restart, refresh edit-log failure, FORCE late commit after a newer same-name job, and credentials/no-leak; - a separately asserted single-node local-path test; it is not evidence for arbitrary distributed mount equivalence; - REST pagination/stats bounds, field NULL/provenance, both TVF SHOW denials, and pre-job rejection of every mutation. Tests use `run-fe-ut.sh`, `run-be-ut.sh`, and `run-regression-test.sh`. Cross-SDK, I/O-recorder, object-storage, and cgroup-dependent tests run in CI or a documented environment with the pinned libraries and required capabilities. ## Appendix E. Primary evidence ### Doris baseline - [Parser index surface](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4#L235-L240) - [`CreateIndexOp` internal serialization](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateIndexOp.java#L73-L89) - [`ShowIndexCommand` current scope](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowIndexCommand.java#L51-L132) - [External-table refresh/edit log](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java#L123-L209) - [FE `lance-core` pin](https://github.com/apache/doris/blob/1d147d8ec65576d3edf4c9ca0b6a36078193e4d2/fe/pom.xml#L337-L337) - [BE `lance-c` pin](https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/thirdparty/vars.sh#L577-L581) ### Pinned Lance sources - [lance-c lifecycle ABI](https://github.com/lance-format/lance-c/blob/v0.1.2/include/lance/lance.h#L494-L538) - [lance-c error model](https://github.com/lance-format/lance-c/blob/v0.1.2/src/error.rs#L4-L143) - [Rust dependencies and `panic=abort`](https://github.com/lance-format/lance-c/blob/v0.1.2/Cargo.toml#L20-L63) - [Java Dataset index APIs](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/src/main/java/org/lance/Dataset.java#L1366-L1438) - [Java logical `IndexDescription`](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/src/main/java/org/lance/index/IndexDescription.java#L25-L102) - [Java physical `Index`](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/src/main/java/org/lance/index/Index.java#L29-L131) - [`getIndexStatistics()` side effect](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/rust/lance/src/index.rs#L1791-L1810) - [Namespace create fields](https://github.com/lance-format/lance-namespace/blob/2e4de1555fb8eed8057484f1f3b47ef11ed88d80/java/lance-namespace-apache-client/docs/CreateTableIndexRequest.md#L8-L24) - [Namespace optional transaction ID](https://github.com/lance-format/lance-namespace/blob/2e4de1555fb8eed8057484f1f3b47ef11ed88d80/java/lance-namespace-apache-client/docs/CreateTableIndexResponse.md#L7-L12) - [Namespace list content](https://github.com/lance-format/lance-namespace/blob/2e4de1555fb8eed8057484f1f3b47ef11ed88d80/java/lance-namespace-apache-client/docs/IndexContent.md#L8-L14) - [Namespace transaction fields](https://github.com/lance-format/lance-namespace/blob/2e4de1555fb8eed8057484f1f3b47ef11ed88d80/java/lance-namespace-apache-client/docs/DescribeTransactionResponse.md#L8-L11) ### Review history - [Original issue](https://github.com/apache/doris/issues/66497) - [Round 1](https://github.com/apache/doris/issues/66497#issuecomment-5199781913) - [Round 2](https://github.com/apache/doris/issues/66497#issuecomment-5205105189) - [4.2 scope correction](https://github.com/apache/doris/issues/66497#issuecomment-5211673675) - [Round 4](https://github.com/apache/doris/issues/66497#issuecomment-5214852990) - [Round 5](https://github.com/apache/doris/issues/66497#issuecomment-5226393984) ## Non-normative implementation notes Likely implementation seams are a neutral parsed index specification, a target-aware external provider branch before internal `catalog.Index` serialization, a master-owned Lance job/target-guard manager, a dedicated BE supervisor, an isolated worker launcher, and an FE authoritative metadata reader. Names and class boundaries may change without changing the contract above. The implementation should reuse existing edit-log/image, external-table refresh, master-only daemon, privilege, and result-set conventions. It must not claim that generic `JobManager` or internal `IndexChangeJob` already supplies the required external one-shot CAS, replay, fence, or executor semantics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
