xiangfu0 opened a new pull request, #17994:
URL: https://github.com/apache/pinot/pull/17994
## Summary
- Introduce a **backend-neutral vector index abstraction**
(`VectorBackendType` enum, `VectorIndexConfigValidator`, dispatch in
`VectorIndexType`)
- Add **IVF_FLAT** as a second ANN backend alongside existing HNSW
(Lucene-based)
- Add **query-time tuning** options: `vectorNprobe`, `vectorExactRerank`,
`vectorMaxCandidates`
- Add **exact scan fallback** when no ANN index exists on a segment
- Pure Java implementation — no JNI, no new native dependencies
## IVF_FLAT Backend
- **Creator**: k-means++ centroid training, vector assignment to inverted
lists, flat serialization (`.vector.ivfflat.index`)
- **Reader**: configurable nprobe search (probe N closest centroids, scan
their lists, return top-K)
- **Distance functions**: L2/EUCLIDEAN, COSINE, INNER_PRODUCT, DOT_PRODUCT
- **Immutable segments only** (no mutable/realtime support in phase 1)
## Runtime Integration
- `VectorSearchParams`: extracts vector query options from `QueryContext`
- `NprobeAware` interface: sets nprobe on IVF_FLAT reader at query time
- `ExactVectorScanFilterOperator`: brute-force fallback when no index exists
- `VectorSimilarityFilterOperator`: enhanced with nprobe dispatch and exact
rerank
- `FilterPlanNode`: graceful fallback instead of exception on missing index
## Backward Compatibility
- Existing HNSW configs work **unchanged** (vectorIndexType defaults to HNSW
when omitted)
- SQL syntax (`VECTOR_SIMILARITY`) is **not modified**
- All 10 existing vector tests pass without changes
- Wire protocol and serialization formats unchanged
## Configuration Examples
**IVF_FLAT:**
```json
{
"name": "embedding",
"encodingType": "RAW",
"indexTypes": ["VECTOR"],
"properties": {
"vectorIndexType": "IVF_FLAT",
"vectorDimension": "128",
"vectorDistanceFunction": "EUCLIDEAN",
"nlist": "64"
}
}
```
**Query-time tuning:**
```sql
SET vectorNprobe=16;
SET vectorExactRerank=true;
SELECT ... WHERE VECTOR_SIMILARITY(embedding, ARRAY[...], 10) > 0
```
## Benchmark Results (10K vectors, 128d, L2)
| Config | Recall@10 | p50 Latency | vs Exact Scan |
|---|---|---|---|
| Exact scan | 1.000 | 2,206 us | baseline |
| nlist=16, nprobe=8 | 0.713 | 390 us | 5.7x faster |
| nlist=16, nprobe=16 | 1.000 | 775 us | 2.8x faster |
| nlist=32, nprobe=16 | 0.752 | 402 us | 5.5x faster |
## Test plan
- [x] `VectorBackendTypeTest` — 6 tests for enum parsing
- [x] `VectorIndexConfigValidatorTest` — 33 tests for config validation
- [x] `IvfFlatVectorIndexTest` — 32 tests (round-trips, search, edge cases,
recall)
- [x] `VectorSearchParamsTest` — 13 tests for query option parsing
- [x] `ExactVectorScanFilterOperatorTest` — 7 tests for exact fallback
- [x] `VectorSimilarityFilterOperatorTest` — 10 tests for nprobe/rerank
- [x] `QueryOptionsUtilsTest` — 10 new tests for vector option parsing
- [x] Existing `VectorConfigTest`, `HnswVectorIndexCreatorTest`,
`VectorIndexTest` — all pass
- [x] All 4 modules compile cleanly (`pinot-segment-spi`,
`pinot-segment-local`, `pinot-core`, `pinot-common`)
- [ ] Integration test with full Pinot cluster (HNSW + IVF_FLAT segments)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
