Gabriel39 opened a new issue, #68394: URL: https://github.com/apache/doris/issues/68394
## Motivation Track Batch Search for Lance vector retrieval under #66340. A batch contains independent query vectors, each producing its own Top-K results. For example, three queries with `top_k=10` return up to 30 rows, with a query index identifying which query produced each result. This is distinct from Multi-vector Search, where several subvectors form one logical query and produce one ranked result set. It is also distinct from issuing multiple concurrent SQL statements or changing the scanner's output batch size. Creating this issue tracks the implementation work; the target release remains to be decided. ## Current status and upstream dependencies Checked against `branch-4.1` at `85c9e0aab94f325907fd3a420f56e00fd5b1380f`: - `vector_search()` accepts one logical query. `TSearchVector.num_vectors` represents the subvectors of one multi-vector query, not independent queries. - The BE reader calls `lance_scanner_nearest()` or `lance_scanner_nearest_multivector()`. Batch input and a query-index result column are not exposed through Doris/lance-c. - The planner inserts one global Top-N above the search TVF to merge split results. Batch Search requires an independent Top-N for each query. - Lance Rust already supports batch queries through [lance-format/lance#6828](https://github.com/lance-format/lance/pull/6828), including in the Lance v11 dependency used by the current Doris patch chain. Basic support does not require the newer shared-scan optimization. - [lance-format/lance#7640](https://github.com/lance-format/lance/pull/7640) adds shared IVF partition scans. Its optimized path has eligibility restrictions; HNSW, refinement, adaptive nprobes, and other unsupported combinations fall back to per-query execution. Exposing batch input alone does not guarantee shared index computation. The parent issue's statement that Lance Scanner accepts only one independent query vector is therefore outdated. The remaining work includes the C binding and Doris integration. ## Proposed initial contract - Search one fixed snapshot, vector column, and dataset per batch. - Accept a matrix of independent single-vector queries using an explicit batch input, tentatively `query_vectors`, mutually exclusive with `query_vector`. Final naming is subject to API review. - Share metric, filter, search options, `top_k`, and `offset` across the batch initially. - Return a zero-based query identifier, tentatively `_query_index`, alongside the existing projected columns and `_distance`. - Apply `top_k` and `offset` independently to each query. A query may return fewer than K rows. Repeated query vectors retain distinct query identifiers. - Preserve existing single-vector and multi-vector behavior. Define filter placement explicitly and retain the distinction between the search filter and an outer SQL WHERE. - Batch input must not imply an unspecified SQL output order. Document how callers can order results by query identifier and distance. ## Implementation work - [ ] FE API and analysis: validate batch shape, dimensions, supported element types, null/non-finite values, reserved-column conflicts, and input size; support prepared-statement binding without reusing stale query data. - [ ] Execution protocol: represent independent query count and payload explicitly, separately from multi-vector subvector count; define compatibility behavior for unsupported FE/BE combinations. - [ ] lance-c: add a batch-nearest C/C++ API that builds the appropriate Arrow query array for Lance and preserves the query-index output. Define buffer ownership, validation, error handling, and cancellation behavior. - [ ] BE reader: pass batch requests, map the query-index virtual column, preserve the pinned snapshot and fragment/index-segment scope, and propagate stream failures and cancellation. - [ ] Planner and merge: produce up to `top_k + offset` candidates per query per split, merge by query identifier, and apply per-query offset/Top-K. Do not replace this with a global `LIMIT batch_size * top_k`. Audit filter, projection, and limit rewrites around the search TVF. - [ ] Resource control: bound batch size, payload bytes, candidate expansion across queries/splits/refinement, internal concurrency, and buffered results; account for memory and release resources on cancellation or errors. - [ ] Dependency integration: update the Doris lance-c dependency/patch chain. Evaluate the shared-IVF-scan optimization separately from basic batch support, including upgrade/backport compatibility and fallback behavior. - [ ] EXPLAIN/Profile: expose query count, execution mode (shared batch or per-query fallback), prefilter construction count/time, unique index partitions loaded where measurable, search time, and result materialization time. Document counter scope and aggregation. - [ ] Documentation and benchmark client: document semantics and limitations, and add a client path that actually submits batches instead of independent SQL requests. ## Acceptance criteria - [ ] FE/BE unit tests and regression tests cover batch size one and multiple queries, duplicate query vectors, empty results, invalid input, prepared-statement reuse, and resource limits. - [ ] Validate each query against independent execution: exact-search results match under a defined tie policy; ANN tests use controlled inputs/options and validate the documented approximate-search semantics and recall. - [ ] Cover multiple fragments, index segments, and BE splits, including offset, deletion visibility, indexed/unindexed data, filters, and projections. No query may lose results because another query consumed a global limit. - [ ] Verify supported index/metric/type combinations and explicit fallback behavior, including HNSW and refinement. Existing single-query and multi-vector tests continue to pass. - [ ] Cancellation and failure tests demonstrate that batch tasks, streams, and memory are released. - [ ] Benchmark several batch sizes and concurrency levels with the same dataset, search options, and comparable recall. Report query-vector throughput, batch latency, amortized time per vector, and peak memory separately; do not interpret batch latency divided by batch size as individual request latency. - [ ] Document measured benefits and limitations. In particular, do not claim the shared IVF scan speedup for HNSW's per-query fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
