u70b3 opened a new pull request, #68957:
URL: https://github.com/apache/doris/pull/68957
### What problem?
The durable lance index job framework (#67235, job-specific parts of #67630)
was removed by #68781 (merged). Per the execution-model direction confirmed in
#66497 (synchronous SQL statement: validate → dispatch resource-isolated work
to one BE → wait → classify → refresh → return), this PR adds the FE-side
synchronous execution path and the BE dispatch contract, building directly on
the post-revert branch-4.1 state.
Issue Number: #66497
### Problem Summary
What lands:
- **Outcome model**: every mutation statement returns exactly one of four
classes — success, confirmed failure, indeterminate outcome (explicit error
stating the mutation may or may not have committed, directing to `SHOW INDEX` /
`lance_index_entries()`; never auto-retried), and
committed-with-refresh-incomplete (a distinct error that is never a build
failure). New error codes 5104–5108.
- **Dispatch contract**: thrift `lance_index_mutate` (single
request/response, single-send, no retry on any path). The BE ships a stub
returning a clean `NOT_IMPLEMENTED_ERROR` — a complete pre-invocation
rejection, classified as confirmed failure. Worker execution lands in a
follow-up PR.
- **Synchronous executor** (master, in-memory only, nothing persisted):
- per (catalog, dataset, normalized index name) active-name entry —
same-name requests fail fast with a typed busy error; the entry is released in
a `finally` once classification and refresh complete;
- bounded admission: dedicated pool (defaults 2 concurrent / 16 queued) —
saturation fails fast;
- classification follows the pinned ABI facts: exact commit-conflict code
→ confirmed failure; any other typed code after dispatch, transport failure,
timeout, or partial response → indeterminate; message text is never parsed;
- one time budget per statement: min(query timeout,
`lance_index_mutation_budget_cap_seconds`, default 300 s) covering admission,
queueing, dispatch, and refresh; pre-dispatch exhaustion is a confirmed failure
("nothing was sent");
- refresh obligations: a confirmed commit invalidates the local table
caches first (bounded retries); failure downgrades the outcome to
committed-refresh-incomplete (5105). A no-op completes its local invalidation
before returning (residual failure is a confirmed-family refresh error, 5107).
A commit-conflict refreshes best-effort with a diagnostic only.
- **Wiring**: `CREATE INDEX` / `CREATE OR REPLACE INDEX` / `DROP INDEX` on
Directory-catalog Lance tables run through the executor on the master after the
preserved admission validation (privilege/schema/parameter/snapshot/IF). The
final authoritative IF preflight re-runs inside the name entry, so a queued
request cannot dispatch on a stale snapshot; an authoritative IF no-op
completes its bounded local cache invalidation before returning. `ALTER TABLE
... ADD/DROP INDEX` and `BUILD INDEX` remain rejected (5101). Forwarding
semantics (`ForwardWithSync`) are unchanged.
- The mutation gate `enable_lance_index_mutation` remains **default off**.
Commits (7): outcome model + error codes 5104–5108 · thrift dispatch
contract + BE stub · synchronous executor · refresh obligations · statement
wiring + suite flips · groovy regression update · review-fix round (adversarial
review: 0 P1 / 3 P2 / 8 P3, all P2 and six P3 closed).
Design readings pending maintainer confirmation (taken where the direction
comment left room): the in-memory active-name entry (no durable fence); the
minimal statement-level budget (cross-host verifiable expiration lands with the
worker PR); a single FE-side admission tier (the BE-side bounded queue lands
with the worker PR); transport failures split at the connection-establishment
boundary (an exception while establishing the connection proves zero payload
bytes were sent → confirmed failure; any failure after the send started →
indeterminate); the refresh path reserves no separate deadline margin beyond
the statement budget; the busy-guard key uses the catalog's normalized dataset
URI as-is.
### Not in this PR
The real isolated worker/supervisor (next PR, reworks the closed
#68668/#68669 assets), domain-termination-proof capacity semantics, distributed
index construction, REST surfaces.
### Validation
- Focused FE UT (`Lance*,ShowIndexCommandTest`, final head): **41 classes,
487 fe-core tests + 5 fe-common, 0 failures / 0 errors / 7 pre-existing
environmental skips** (baseline before this PR: 439 tests / 38 classes).
Includes 25+ executor tests, 12 refresh tests, 15 outcome tests, 5 contract
tests, 4 real-socket dispatcher tests, and the flipped admission/command suites.
- Full FE UT (final head, clean run): **10,459 fe-core tests, 19 failures +
2 errors confined to the 14 known environmental classes
(timezone/network/native), matching the pre-PR baseline class-for-class — zero
new failures**; fe-common green
(PropertySchemaTest/AuthenticationPluginManagerTest excluded per
known-environment convention).
- BE: stub-only change following the existing pure-virtual stub pattern;
compile verified by CI.
- Regression groovy: 3 flipped gate-on cases (5108 + stub message),
read-face cross-validation (TVF/SHOW unchanged after confirmed failures),
non-Lance negatives, BUILD INDEX/ALTER ADD-DROP INDEX still rejected; groovy
sources compile-verified locally (FileSystemCompiler), runtime verification via
pipeline.
### Release note
None (feature gated off by default; mutation surface unchanged while
`enable_lance_index_mutation=false`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
