This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new d4920972ab [index] Update paimon-vindex to 0.3.0 (#8919)
d4920972ab is described below
commit d4920972ab322fa65bdce37bc61062fb1adefd4d
Author: Jingsong Lee <[email protected]>
AuthorDate: Thu Jul 30 13:10:36 2026 +0800
[index] Update paimon-vindex to 0.3.0 (#8919)
---
docs/docs/flink/procedures.md | 2 +-
docs/docs/multimodal-table/global-index.mdx | 2 +-
.../global-index/hybrid-search.mdx | 2 +-
docs/docs/multimodal-table/global-index/vector.mdx | 113 ++++++++++-----------
docs/docs/primary-key-table/global-index.mdx | 2 +-
docs/docs/program-api/file-cache.mdx | 2 +-
paimon-python/dev/requirements-dev.txt | 2 +-
paimon-python/dev/run_mixed_tests.sh | 6 +-
.../vindex/vindex_vector_global_index_reader.py | 48 ++++++---
.../vindex/vindex_vector_index_writer.py | 104 +++++++++++++++++--
.../pypaimon/tests/global_index_build_test.py | 83 +++++++++++++--
.../pypaimon/tests/vindex_vector_index_test.py | 51 +++++++---
paimon-python/setup.py | 2 +-
paimon-vector/pom.xml | 2 +-
...java => DiskAnnVectorGlobalIndexerFactory.java} | 6 +-
...y.java => IvfRqVectorGlobalIndexerFactory.java} | 6 +-
...y.java => IvfSqVectorGlobalIndexerFactory.java} | 6 +-
.../index/NativeVectorGlobalIndexReader.java | 64 ++++++++----
.../index/NativeVectorGlobalIndexWriter.java | 17 +++-
.../vector/index/NativeVectorGlobalIndexer.java | 2 +-
.../index/NativeVectorGlobalIndexerFactory.java | 38 ++++++-
.../vector/index/NativeVectorIndexLoader.java | 112 --------------------
....apache.paimon.globalindex.GlobalIndexerFactory | 5 +-
.../vector/index/NativeVectorGlobalIndexTest.java | 88 +++++++++++++---
.../NativeVectorGlobalIndexerFactoryTest.java | 61 +++++++++--
25 files changed, 539 insertions(+), 287 deletions(-)
diff --git a/docs/docs/flink/procedures.md b/docs/docs/flink/procedures.md
index 927bfe231b..d51423128c 100644
--- a/docs/docs/flink/procedures.md
+++ b/docs/docs/flink/procedures.md
@@ -1060,7 +1060,7 @@ All available procedures are listed below.
To create a global index on a table for accelerating queries.
Arguments:
<li>table(required): the target table identifier.</li>
<li>index_column(required): the column name to build index on.</li>
- <li>index_type(required): the type of global index, supported
types include 'btree', 'bitmap', 'ivf-flat', 'ivf-pq', 'ivf-hnsw-flat',
'ivf-hnsw-sq', 'full-text'.</li>
+ <li>index_type(required): the type of global index, supported
types include 'btree', 'bitmap', 'ivf-flat', 'ivf-pq', 'ivf-sq', 'ivf-rq',
'diskann', and 'full-text'.</li>
<li>partitions(optional): partition filter for selective index
creation.</li>
<li>options(optional): additional dynamic options for index
creation.</li>
</td>
diff --git a/docs/docs/multimodal-table/global-index.mdx
b/docs/docs/multimodal-table/global-index.mdx
index 54aa978f1a..26ca42cc14 100644
--- a/docs/docs/multimodal-table/global-index.mdx
+++ b/docs/docs/multimodal-table/global-index.mdx
@@ -158,7 +158,7 @@ added_files = table.create_global_index(
</Tabs>
PyPaimon global index build currently supports single-column BTree indexes,
-single-column Bitmap indexes, single-column paimon-vindex IVF vector indexes,
+single-column Bitmap indexes, single-column paimon-vindex vector indexes,
and single-column full-text indexes on tables with row tracking enabled.
Drop index files:
diff --git a/docs/docs/multimodal-table/global-index/hybrid-search.mdx
b/docs/docs/multimodal-table/global-index/hybrid-search.mdx
index 4d8192e3ab..f767488927 100644
--- a/docs/docs/multimodal-table/global-index/hybrid-search.mdx
+++ b/docs/docs/multimodal-table/global-index/hybrid-search.mdx
@@ -79,7 +79,7 @@ The second argument is an array of vector route configs
created by `named_struct
| `query_vector` | Yes | N/A | Query vector for this route. |
| `limit` | No | Final limit | Top K results to retrieve from this vector
column before ranking. |
| `weight` | No | `1.0` | Weight for this route when ranking results. |
-| `options` | No | Empty map | Route-specific vector search options such as
`ivf.nprobe` and `hnsw.ef_search`. |
+| `options` | No | Empty map | Route-specific vector search options such as
`ivf.nprobe` and `diskann.l_search`. |
The third argument is an array of full-text route configs created by
`named_struct`:
diff --git a/docs/docs/multimodal-table/global-index/vector.mdx
b/docs/docs/multimodal-table/global-index/vector.mdx
index d29d076769..aec9ab0cef 100644
--- a/docs/docs/multimodal-table/global-index/vector.mdx
+++ b/docs/docs/multimodal-table/global-index/vector.mdx
@@ -36,19 +36,31 @@ Supported vector index types:
|---|---|
| `ivf-flat` | IVF index with flat vector storage. |
| `ivf-pq` | IVF index with product quantization. |
-| `ivf-hnsw-flat` | IVF index with HNSW flat quantizer. |
-| `ivf-hnsw-sq` | IVF index with HNSW scalar quantizer. |
-| `lumina` | Lumina DiskANN-based vector index. |
+| `ivf-sq` | IVF index with 8-bit scalar-quantized residuals. |
+| `ivf-rq` | IVF index with rotated residual quantization. |
+| `diskann` | paimon-vindex DiskANN index with graph traversal and persisted
rerank vectors. |
Choose the index type based on the trade-off you want:
| Index Type | Best For |
|---|---|
| `ivf-flat` | Highest recall among IVF variants when storage and memory are
acceptable. |
-| `ivf-pq` | Smaller index files and a balanced recall, latency, and storage
trade-off. |
-| `ivf-hnsw-flat` | Better recall inside IVF partitions with raw vector
storage. |
-| `ivf-hnsw-sq` | HNSW search quality with scalar quantization to reduce index
size. |
-| `lumina` | Large-scale ANN search with DiskANN graph indexing and
configurable `rawf32`, `sq8`, or `pq` encodings. |
+| `ivf-pq` | The smallest IVF files when stronger quantization loss is
acceptable. |
+| `ivf-sq` | High compact-index throughput with one byte per vector dimension.
|
+| `ivf-rq` | Higher compact-IVF recall when additional scan work is
acceptable. |
+| `diskann` | High-recall immutable collections served from local SSD or a
complete local cache. |
+
+See the [paimon-vindex
documentation](https://paimon.apache.org/docs/vector-index/) for its index
+selection guidance, storage architecture, and native API details.
+
+:::warning Upgrading paimon-vindex indexes to 0.3.0
+
+paimon-vindex 0.3.0 does not read the experimental `ivf-hnsw-flat` and
`ivf-hnsw-sq` files written
+by 0.2.x. Rebuild those indexes as `ivf-flat`, `ivf-pq`, `ivf-sq`, `ivf-rq`,
or `diskann` before
+removing the old runtime. Keep the source vectors or a 0.2-compatible index
copy until the upgrade
+is accepted.
+
+:::
## Build Vector Index
@@ -62,15 +74,15 @@ CALL sys.create_global_index(
table => 'db.my_table',
index_column => 'embedding',
index_type => 'ivf-pq',
- options => 'ivf-pq.distance.metric=cosine,ivf-pq.nlist=256,ivf-pq.pq.m=16'
+ options =>
'ivf-pq.distance.metric=cosine,ivf-pq.nlist=256,ivf-pq.pq.code-ratio=0.0625'
);
--- Create Lumina DiskANN vector index on 'embedding' column
+-- Create a paimon-vindex DiskANN vector index
CALL sys.create_global_index(
table => 'db.my_table',
index_column => 'embedding',
- index_type => 'lumina',
- options =>
'lumina.index.dimension=768,lumina.distance.metric=l2,lumina.encoding.type=sq8'
+ index_type => 'diskann',
+ options =>
'diskann.dimension=768,diskann.distance.metric=l2,diskann.build-preset=balanced'
);
```
@@ -89,7 +101,7 @@ added_files = table.create_global_index(
"ivf-pq.dimension": "768",
"ivf-pq.distance.metric": "cosine",
"ivf-pq.nlist": "256",
- "ivf-pq.pq.m": "16",
+ "ivf-pq.pq.code-ratio": "0.0625",
},
)
print(added_files)
@@ -111,22 +123,16 @@ added_files = table.create_global_index(
print(added_files)
```
-PyPaimon vector index build currently supports the paimon-vindex IVF
identifiers:
-`ivf-flat`, `ivf-pq`, `ivf-hnsw-flat`, and `ivf-hnsw-sq`. Use SQL to build
-Lumina indexes. Install `paimon-vindex` or `pypaimon[vindex]` before building
-or querying IVF vector indexes from Python.
+PyPaimon vector index build supports the paimon-vindex 0.3.0 identifiers
`ivf-flat`, `ivf-pq`,
+`ivf-sq`, `ivf-rq`, and `diskann`. Install `paimon-vindex==0.3.0` or
`pypaimon[vindex]` before
+building or querying paimon-vindex indexes from Python.
</TabItem>
</Tabs>
-Use `index_type => 'lumina'` for new Lumina indexes. The legacy
`lumina-vector-ann` identifier is
-kept only so existing tables can still load old indexes.
-
For `ARRAY<FLOAT>` vector columns, specify the vector dimension with
`<index-type>.dimension` for
-IVF indexes or `lumina.index.dimension` for Lumina indexes. For
`VECTOR<FLOAT>` columns, Paimon uses
-the dimension from the column type. When `lumina.index.dimension` is
explicitly set for a
-`VECTOR<FLOAT>` column, it must match the vector type length.
+paimon-vindex indexes. For `VECTOR<FLOAT>` columns, Paimon uses the dimension
from the column type.
## Drop Vector Index
@@ -170,41 +176,36 @@ print(matched_files)
</Tabs>
-Supported IVF vector index options:
+Supported paimon-vindex options:
| Option | Default | Description |
|---|---|---|
| `<index-type>.dimension` | `128` | Vector dimension for `ARRAY<FLOAT>`
columns. Ignored for `VECTOR<FLOAT>` columns. |
| `<index-type>.distance.metric` | `inner_product` | Distance metric.
Supported values: `l2`, `cosine`, `inner_product`. |
| `<index-type>.train.sample-ratio` | `1.0` | Ratio of vectors sampled for
native index training. Must be greater than `0` and less than or equal to `1`.
Lower values reduce training memory and build cost, but may reduce index
quality. |
-| `<index-type>.nlist` | `256` | Number of IVF clusters used during index
build. Higher values create more partitions and can improve recall for large
datasets, but may increase build cost. |
-| `<index-type>.pq.m` | `16` | Number of PQ sub-vectors for `ivf-pq`. The
vector dimension must be divisible by this value. Higher values usually improve
recall with larger index files. |
-| `<index-type>.pq.use-opq` | `false` | Whether to enable OPQ for `ivf-pq`. |
-| `<index-type>.hnsw.m` | `20` | HNSW graph out-degree for `ivf-hnsw-flat` and
`ivf-hnsw-sq`. |
-| `<index-type>.hnsw.ef-construction` | `150` | HNSW construction search width
for `ivf-hnsw-flat` and `ivf-hnsw-sq`. |
-| `<index-type>.hnsw.max-level` | `7` | Maximum HNSW level for `ivf-hnsw-flat`
and `ivf-hnsw-sq`. |
+| `<index-type>.nlist` | Automatic | Number of clusters for the four IVF
types. When omitted, paimon-vindex resolves it from the shard's non-null vector
count. |
+| `<index-type>.pq.code-ratio` | `0.0625` | Relative PQ-code budget for
`ivf-pq` and `diskann`. |
+| `<index-type>.pq.m` | Automatic | Expert override for the PQ sub-vector
count used by `ivf-pq` and `diskann`. |
+| `ivf-pq.pq.use-opq` | Automatic | Explicitly enables or disables OPQ.
Without an explicit value, a `target-recall` of at least `0.9` enables it. |
+| `ivf-rq.rq.bits` | `4` | Persisted IVF-RQ residual width. Supported values
are `1` through `8`; changing it requires rebuilding the index. |
+| `<index-type>.target-recall` | Not set | Build-policy hint used by `ivf-pq`
and `diskann`. Validate the resulting recall on held-out queries. |
+| `<index-type>.max-bytes-per-vector` | Not set | Storage objective and
conservative preflight bound for `ivf-pq`, `ivf-rq`, and `diskann`. |
-Supported Lumina vector index options:
+Additional paimon-vindex DiskANN build options:
| Option | Default | Description |
|---|---|---|
-| `lumina.index.dimension` | `128` | Vector dimension for `ARRAY<FLOAT>`
columns. For `VECTOR<FLOAT>` columns, an explicitly configured value must match
the type length. |
-| `lumina.distance.metric` | `inner_product` | Distance metric. Supported
values: `l2`, `cosine`, `inner_product`. |
-| `lumina.index.type` | `diskann` | Lumina index type. Currently supports
DiskANN. |
-| `lumina.encoding.type` | `pq` | Vector encoding type. Supported values:
`rawf32`, `sq8`, `pq`. |
-| `lumina.pretrain.sample_ratio` | `0.2` | Sample ratio used for pretraining. |
-| `lumina.diskann.build.ef_construction` | `1024` | Size of the dynamic
candidate list during DiskANN graph construction. |
-| `lumina.diskann.build.neighbor_count` | `64` | Maximum number of neighbors
per node in the DiskANN graph. |
-| `lumina.diskann.build.thread_count` | `32` | Number of threads used for
DiskANN index building. |
-| `lumina.diskann.search.list_size` | unset; search uses `max(1.5x top_k, 16)`
| Default DiskANN search list size used when no query value is supplied. |
-| `lumina.diskann.search.beam_width` | `4` | Beam width for DiskANN search. |
-| `lumina.encoding.pq.m` | `64` | Number of sub-quantizers for PQ encoding. It
is capped to the vector dimension when larger than the dimension. |
-| `lumina.search.parallel_number` | `5` | Parallel number for Lumina search. |
-
-Lumina PQ encoding does not support the `cosine` distance metric. Use `rawf32`
or `sq8` encoding
-with `cosine`, or use `l2` or `inner_product` with `pq`.
-
-The Lumina native library is currently available only on x86_64 (AMD64)
architecture.
+| `diskann.build-preset` | `balanced` | Coherent `fast_build`, `balanced`, or
`high_recall` build policy. |
+| `diskann.deployment-profile` | Not set | Deployment objective used to choose
a storage layout. |
+| `diskann.pq.bits` | `8` | Resident PQ-code width. Supported values are `4`
and `8`. |
+| `diskann.max-degree` | `64` | Maximum graph out-degree. |
+| `diskann.build-search-list-size` | `max(100, max-degree)` | Candidate width
during graph construction. |
+| `diskann.alpha` | `1.2` | Robust-prune threshold. |
+| `diskann.seed` | `42` | Reproducible initialization and build-order seed. |
+| `diskann.memory-budget-bytes` | `8 GiB` | Internal graph-build memory
estimate used to select normal or sharded construction. |
+| `diskann.storage-layout` | `auto` | Explicit `compact` or `interleaved`
layout override. |
+| `diskann.raw-vector-encoding` | `auto` | Explicit `f32` or `f16` persisted
rerank-vector encoding. |
+| `diskann.build-distance` | `auto` | Explicit product-quantized or
full-precision build traversal override. |
## Per-Field Options
@@ -278,27 +279,19 @@ catalog.create_table("db.my_table", schema,
ignore_if_exists=False)
With the properties above, `title_embedding` is indexed with `nlist=256` while
`image_embedding`
uses `nlist=512` and trains with half of the non-null vectors.
-Lumina uses the same field-level convention. For example,
`fields.image_embedding.distance.metric`
-overrides `lumina.distance.metric` for `image_embedding`, and
-`fields.image_embedding.index.dimension` overrides `lumina.index.dimension`.
-
## Vector Search
Search-time options are passed with each vector search request:
| Option | Default | Description |
|---|---|---|
-| `ivf.nprobe` | `16` | Number of IVF clusters to probe during search. Higher
values usually improve recall but increase latency. |
-| `ivf.refine_factor` | Disabled | Retrieves `top_k * refine_factor` IVF
candidates and reranks them with the original vectors stored in the Paimon
table. It is disabled by default for every IVF variant and is most useful for
compressed indexes such as `ivf-pq` and `ivf-hnsw-sq` when recall is more
important than latency. |
-| `hnsw.ef_search` | `0` | HNSW search width during search. Higher values
usually improve recall but increase latency. `0` uses the native library
default. |
-| `diskann.search.list_size` | `max(1.5x top_k, 16)` | Lumina DiskANN search
list size. Higher values usually improve recall but increase latency. |
-| `diskann.search.beam_width` | `4` | Lumina DiskANN search beam width. |
-| `search.parallel_number` | `5` | Lumina search parallel number. |
+| `ivf.nprobe` | Automatic | Explicit number of IVF clusters to probe. When
omitted, paimon-vindex derives the width from the index, `top_k`, and filter
selectivity. |
+| `ivf.refine_factor` | Disabled | Retrieves `top_k * refine_factor` IVF
candidates and reranks them with the original vectors stored in the Paimon
table. It is most useful for compressed indexes such as `ivf-pq`, `ivf-sq`, and
`ivf-rq` when recall is more important than latency. |
+| `diskann.l_search` | Automatic | paimon-vindex DiskANN graph candidate
width. The automatic value uses calibration when available, otherwise `max(100,
2 * top_k)`. |
Use the same distance metric at build time and query time. Search options can
be passed per query,
-so you can use a larger `ivf.nprobe` or `hnsw.ef_search` for higher recall
queries and a smaller
-value for latency-sensitive queries. Lumina query-time options use the native
keys shown above; when
-the same options are configured as table or index options, use the `lumina.`
prefix.
+so you can use a larger `ivf.nprobe` or `diskann.l_search` for higher recall
queries and a smaller
+value for latency-sensitive queries. Do not set both in one query.
`ivf.refine_factor` can also be configured with `refine_factor`,
`rerank_factor`, and hyphenated
spellings such as `ivf.refine-factor`. Setting `ivf.refine_factor=1` still
performs the raw-vector
diff --git a/docs/docs/primary-key-table/global-index.mdx
b/docs/docs/primary-key-table/global-index.mdx
index 04008c2db5..db6ebe00d1 100644
--- a/docs/docs/primary-key-table/global-index.mdx
+++ b/docs/docs/primary-key-table/global-index.mdx
@@ -225,7 +225,7 @@ schema validation.
| Option | Default | Description |
|---|---|---|
| `pk-vector.index.columns` | Not set | Vector column to index. Exactly one
vector column is currently supported. |
-| `fields.<column>.pk-vector.index.type` | Required | ANN implementation, such
as `ivf-flat`, `ivf-pq`, `ivf-hnsw-flat`, `ivf-hnsw-sq`, or `lumina`. |
+| `fields.<column>.pk-vector.index.type` | Required | ANN implementation, such
as `ivf-flat`, `ivf-pq`, `ivf-sq`, `ivf-rq`, or `diskann`. |
| `fields.<column>.pk-vector.distance.metric` | `inner_product` | Distance
metric: `l2`, `cosine`, or `inner_product`. |
| `fields.<column>.pk-vector.index.options` | Not set | JSON object containing
build options for the selected ANN implementation. |
| `pk-full-text.index.columns` | Not set | Character column to index. Exactly
one full-text column is currently supported. |
diff --git a/docs/docs/program-api/file-cache.mdx
b/docs/docs/program-api/file-cache.mdx
index cbed97f57c..eff163ada8 100644
--- a/docs/docs/program-api/file-cache.mdx
+++ b/docs/docs/program-api/file-cache.mdx
@@ -40,7 +40,7 @@ The cache classifies files by type. By default, only `meta`
and `global-index` t
| File Type | Config Name | Examples | Default Cached |
|-----------|-------------|----------|----------------|
| META | meta | snapshot, schema, manifest, statistics, tag | Yes |
-| GLOBAL_INDEX | global-index | BTree, Lumina, full-text index files | Yes |
+| GLOBAL_INDEX | global-index | BTree and full-text index files | Yes |
| BUCKET_INDEX | bucket-index | Hash, deletion vector index files | No |
| DATA | data | Data files (ORC, Parquet, etc.) | No |
| FILE_INDEX | file-index | Data-file level bloom filter, bitmap | No |
diff --git a/paimon-python/dev/requirements-dev.txt
b/paimon-python/dev/requirements-dev.txt
index 4092c5158e..ed4181ffed 100644
--- a/paimon-python/dev/requirements-dev.txt
+++ b/paimon-python/dev/requirements-dev.txt
@@ -33,6 +33,6 @@ datafusion>=52; python_version >= "3.10"
# Lumina vector search (optional, for lumina index tests)
lumina-data>=0.1.0
# paimon-vindex vector search (optional, for vindex index tests)
-paimon-vindex==0.1.0; python_version >= "3.9"
+paimon-vindex==0.3.0; python_version >= "3.9"
# paimon-ftindex full-text search (optional, for full-text index tests)
paimon-ftindex==0.1.0; python_version >= "3.8"
diff --git a/paimon-python/dev/run_mixed_tests.sh
b/paimon-python/dev/run_mixed_tests.sh
index 40b7b22816..057e056591 100755
--- a/paimon-python/dev/run_mixed_tests.sh
+++ b/paimon-python/dev/run_mixed_tests.sh
@@ -660,14 +660,14 @@ ensure_paimon_vindex() {
fi
echo "Installing Python paimon-vindex dependency..."
- if python -m pip install 'paimon-vindex==0.1.0'; then
+ if python -m pip install 'paimon-vindex==0.3.0'; then
return 0
fi
echo -e "${YELLOW}Direct pip install failed; installing paimon-vindex into
a temporary target directory...${NC}"
local target_dir="${TMPDIR:-/tmp}/paimon-vindex-site"
rm -rf "$target_dir"
- if python -m pip install --target "$target_dir" 'paimon-vindex==0.1.0';
then
+ if python -m pip install --target "$target_dir" 'paimon-vindex==0.3.0';
then
export PYTHONPATH="$target_dir:${PYTHONPATH:-}"
return 0
fi
@@ -675,7 +675,7 @@ ensure_paimon_vindex() {
if python -c "import numpy" >/dev/null 2>&1; then
echo -e "${YELLOW}Dependency install failed but numpy is already
available; retrying paimon-vindex without dependencies...${NC}"
rm -rf "$target_dir"
- if python -m pip install --target "$target_dir" --no-deps
'paimon-vindex==0.1.0'; then
+ if python -m pip install --target "$target_dir" --no-deps
'paimon-vindex==0.3.0'; then
export PYTHONPATH="$target_dir:${PYTHONPATH:-}"
return 0
fi
diff --git
a/paimon-python/pypaimon/globalindex/vindex/vindex_vector_global_index_reader.py
b/paimon-python/pypaimon/globalindex/vindex/vindex_vector_global_index_reader.py
index 4006c7d534..98aa34647a 100644
---
a/paimon-python/pypaimon/globalindex/vindex/vindex_vector_global_index_reader.py
+++
b/paimon-python/pypaimon/globalindex/vindex/vindex_vector_global_index_reader.py
@@ -26,12 +26,10 @@ from pypaimon.common.file_io import pread, supports_pread
from pypaimon.globalindex.global_index_reader import GlobalIndexReader,
_completed_future
from pypaimon.globalindex.vector_search_result import
DictBasedScoredIndexResult
-VINDEX_IDENTIFIERS = ("ivf-flat", "ivf-pq", "ivf-hnsw-flat", "ivf-hnsw-sq")
+VINDEX_IDENTIFIERS = ("ivf-flat", "ivf-pq", "ivf-sq", "ivf-rq", "diskann")
NPROBE_PARAMETER = "ivf.nprobe"
-EF_SEARCH_PARAMETER = "hnsw.ef_search"
-DEFAULT_NPROBE = 16
-DEFAULT_EF_SEARCH = 0
+L_SEARCH_PARAMETER = "diskann.l_search"
class PaimonVindexInput:
@@ -67,6 +65,7 @@ class VindexVectorGlobalIndexReader(GlobalIndexReader):
self._index_input = None
self._reader = None
self._metadata = None
+ self._search_params_type = None
self._load_lock = threading.Lock()
def visit_vector_search(self, vector_search):
@@ -97,21 +96,18 @@ class VindexVectorGlobalIndexReader(GlobalIndexReader):
return [None] * n
options = query_options or {}
- nprobe = _int_parameter(options, NPROBE_PARAMETER, DEFAULT_NPROBE)
- ef_search = _int_parameter(options, EF_SEARCH_PARAMETER,
DEFAULT_EF_SEARCH)
+ params = _search_params(self._search_params_type, options, effective_k)
filter_bytes = _filter_bytes(include_row_ids)
if n == 1:
ids, distances = self._reader.search(
- queries[0], effective_k, nprobe, ef_search,
filter_bytes=filter_bytes)
+ queries[0], params, filter_bytes=filter_bytes)
return [_result_from_scores(ids, distances, self._metadata.metric)]
if hasattr(self._reader, "search_batch"):
batch_result = self._reader.search_batch(
np.ascontiguousarray(queries),
- effective_k,
- nprobe,
- ef_search,
+ params,
filter_bytes=filter_bytes,
)
return _batch_results(batch_result, n, effective_k,
self._metadata.metric)
@@ -119,7 +115,7 @@ class VindexVectorGlobalIndexReader(GlobalIndexReader):
results = []
for query in queries:
ids, distances = self._reader.search(
- query, effective_k, nprobe, ef_search,
filter_bytes=filter_bytes)
+ query, params, filter_bytes=filter_bytes)
results.append(_result_from_scores(ids, distances,
self._metadata.metric))
return results
@@ -160,24 +156,29 @@ class VindexVectorGlobalIndexReader(GlobalIndexReader):
return
try:
- from paimon_vindex import VectorIndexReader
+ from paimon_vindex import SearchParams, VectorIndexReader
except ImportError as e:
raise ImportError(
"paimon-vindex is required to read vindex vector indexes. "
- "Install paimon-vindex==0.1.0 or pypaimon[vindex].") from e
+ "Install paimon-vindex==0.3.0 or pypaimon[vindex].") from e
file_path = (self._io_meta.external_path
if self._io_meta.external_path
else os.path.join(self._index_path,
self._io_meta.file_name))
stream = self._file_io.new_input_stream(file_path)
+ reader = None
try:
index_input = PaimonVindexInput(stream)
reader = VectorIndexReader(index_input)
self._metadata = reader.metadata()
+ reader.optimize_for_search()
self._index_input = index_input
self._reader = reader
+ self._search_params_type = SearchParams
self._stream = stream
except Exception:
+ if reader is not None:
+ reader.close()
stream.close()
raise
@@ -262,12 +263,27 @@ def _convert_distance_to_score(distance, metric):
raise ValueError("Unknown vector search metric: %s" % metric)
-def _int_parameter(options, key, default_value):
+def _search_params(search_params_type, options, top_k):
+ nprobe = _int_parameter(options, NPROBE_PARAMETER)
+ l_search = _int_parameter(options, L_SEARCH_PARAMETER)
+ if nprobe is not None and l_search is not None:
+ raise ValueError(
+ "Cannot set both '%s' and '%s'."
+ % (NPROBE_PARAMETER, L_SEARCH_PARAMETER)
+ )
+ if nprobe is not None:
+ return search_params_type.ivf(top_k=top_k, nprobe=nprobe)
+ if l_search is not None:
+ return search_params_type.diskann(top_k=top_k, l_search=l_search)
+ return search_params_type.automatic(top_k=top_k)
+
+
+def _int_parameter(options, key):
value = options.get(key)
if value is None:
- return default_value
+ return None
try:
return int(value)
- except ValueError as e:
+ except (TypeError, ValueError) as e:
raise ValueError(
"Invalid value for '%s': %s. Must be an integer." % (key, value))
from e
diff --git
a/paimon-python/pypaimon/globalindex/vindex/vindex_vector_index_writer.py
b/paimon-python/pypaimon/globalindex/vindex/vindex_vector_index_writer.py
index 2a6ed8670e..2b1f22b697 100644
--- a/paimon-python/pypaimon/globalindex/vindex/vindex_vector_index_writer.py
+++ b/paimon-python/pypaimon/globalindex/vindex/vindex_vector_index_writer.py
@@ -54,6 +54,8 @@ class VindexVectorIndexWriter:
self._index_type = index_type
self._native_options = native_options(
data_type, options, index_type, field_name)
+ self._train_sample_ratio = train_sample_ratio(
+ options, index_type, field_name)
self._dimension = int(self._native_options["dimension"])
self._row_count = 0
self._vector_count = 0
@@ -96,11 +98,11 @@ class VindexVectorIndexWriter:
try:
import numpy as np
- from paimon_vindex import VectorIndexWriter
+ from paimon_vindex import VectorIndexTrainer, VectorIndexWriter
except ImportError as e:
raise ImportError(
"paimon-vindex is required to build vindex vector indexes.
"
- "Install paimon-vindex==0.1.0 or pypaimon[vindex].") from e
+ "Install paimon-vindex==0.3.0 or pypaimon[vindex].") from e
self._close_temp_files()
self._file_io.check_or_mkdirs(self._index_path)
@@ -109,12 +111,19 @@ class VindexVectorIndexWriter:
dtype=np.float32,
count=self._vector_count * self._dimension,
).reshape(self._vector_count, self._dimension)
- with VectorIndexWriter(self._native_options) as writer:
- writer.train(vectors)
+ training_vectors = _sample_training_vectors(
+ np, vectors, self._train_sample_ratio)
+ training = VectorIndexTrainer.train(
+ self._training_options(), training_vectors)
+ try:
+ del training_vectors
del vectors
- self._add_vectors_in_batches(np, writer)
- with self._file_io.new_output_stream(file_path) as
output_stream:
- writer.write(output_stream)
+ with VectorIndexWriter(training) as writer:
+ self._add_vectors_in_batches(np, writer)
+ with self._file_io.new_output_stream(file_path) as
output_stream:
+ writer.write(output_stream)
+ finally:
+ training.close()
except Exception:
self._file_io.delete_quietly(file_path)
raise
@@ -126,6 +135,17 @@ class VindexVectorIndexWriter:
def _file_path(self) -> str:
return "%s/%s" % (self._index_path, self.file_name)
+ def _training_options(self) -> Dict[str, str]:
+ options = dict(self._native_options)
+ nlist = options.get("nlist")
+ if (
+ options.get("index.type", "").startswith("ivf_")
+ and nlist in (None, "auto")
+ and "expected-vector-count" not in options
+ ):
+ options["expected-vector-count"] = str(self._vector_count)
+ return options
+
def close(self) -> None:
if not self._closed:
self._closed = True
@@ -207,9 +227,35 @@ def native_options(
result["index.type"] = index_type.replace('-', '_')
result["dimension"] = str(_dimension(data_type, result, index_type))
+ result.setdefault("metric", "inner_product")
return result
+def train_sample_ratio(
+ options: Mapping[str, object], index_type: str, field_name: str
+) -> float:
+ field_key = "fields.%s.train.sample-ratio" % field_name
+ index_key = "%s.train.sample-ratio" % index_type
+ key = field_key if field_key in options else index_key
+ if key not in options:
+ return 1.0
+
+ value = options[key]
+ try:
+ ratio = float(value)
+ except (TypeError, ValueError) as e:
+ raise ValueError(
+ "Invalid value for '%s': %s. Must be greater than 0 and less "
+ "than or equal to 1." % (key, value)
+ ) from e
+ if not math.isfinite(ratio) or ratio <= 0 or ratio > 1:
+ raise ValueError(
+ "Invalid value for '%s': %s. Must be greater than 0 and less "
+ "than or equal to 1." % (key, value)
+ )
+ return ratio
+
+
def validate_vector_type(data_type: DataType) -> None:
if isinstance(data_type, VectorType):
element_type = data_type.element
@@ -237,14 +283,40 @@ def _native_option_key(option_key: str) -> Optional[str]:
return "metric"
if option_key in (
"nlist",
+ "expected-vector-count",
"pq.m",
- "hnsw.m",
- "hnsw.ef-construction",
- "hnsw.max-level",
+ "pq.code-ratio",
+ "pq.bits",
+ "rq.bits",
+ "target-recall",
+ "max-bytes-per-vector",
+ "deployment-profile",
):
return option_key
if option_key in ("pq.use-opq", "use-opq"):
return "use-opq"
+ diskann_options = {
+ "build-preset": "diskann.build-preset",
+ "diskann.build-preset": "diskann.build-preset",
+ "max-degree": "diskann.max-degree",
+ "diskann.max-degree": "diskann.max-degree",
+ "build-search-list-size": "diskann.build-search-list-size",
+ "diskann.build-search-list-size": "diskann.build-search-list-size",
+ "alpha": "diskann.alpha",
+ "diskann.alpha": "diskann.alpha",
+ "seed": "diskann.seed",
+ "diskann.seed": "diskann.seed",
+ "memory-budget-bytes": "diskann.memory-budget-bytes",
+ "diskann.memory-budget-bytes": "diskann.memory-budget-bytes",
+ "storage-layout": "diskann.storage-layout",
+ "diskann.storage-layout": "diskann.storage-layout",
+ "raw-vector-encoding": "diskann.raw-vector-encoding",
+ "diskann.raw-vector-encoding": "diskann.raw-vector-encoding",
+ "build-distance": "diskann.build-distance",
+ "diskann.build-distance": "diskann.build-distance",
+ }
+ if option_key in diskann_options:
+ return diskann_options[option_key]
return None
@@ -271,6 +343,18 @@ def _is_float_type(data_type: DataType) -> bool:
)
+def _sample_training_vectors(np, vectors, sample_ratio: float):
+ vector_count = vectors.shape[0]
+ train_count = max(1, min(vector_count, int(math.ceil(
+ vector_count * sample_ratio))))
+ if train_count == vector_count:
+ return vectors
+ indexes = (
+ np.arange(train_count, dtype=np.int64) * vector_count // train_count
+ )
+ return np.ascontiguousarray(vectors[indexes])
+
+
def _materialize_vector(
value, dimension: int, relative_row_id: int
) -> List[float]:
diff --git a/paimon-python/pypaimon/tests/global_index_build_test.py
b/paimon-python/pypaimon/tests/global_index_build_test.py
index d03eb4b2f0..186270fcb7 100644
--- a/paimon-python/pypaimon/tests/global_index_build_test.py
+++ b/paimon-python/pypaimon/tests/global_index_build_test.py
@@ -41,7 +41,9 @@ from
pypaimon.globalindex.full_text.native_full_text_index_writer import (
)
from pypaimon.globalindex.vindex.vindex_vector_index_writer import (
VindexVectorIndexWriter,
+ _sample_training_vectors,
native_options,
+ train_sample_ratio,
)
from pypaimon.globalindex.data_evolution_global_index_scanner import
DataEvolutionGlobalIndexScanner
from pypaimon.index.index_file_handler import IndexFileHandler
@@ -78,20 +80,35 @@ class _FakeSplit:
self.raw_convertible = False
+class _FakeVectorIndexTraining:
+
+ def __init__(self, options, data):
+ self.options = dict(options)
+ self.trained = data.tolist()
+ self.closed = False
+
+ def close(self):
+ self.closed = True
+
+
+class _FakeVectorIndexTrainer:
+
+ @classmethod
+ def train(cls, options, data):
+ return _FakeVectorIndexTraining(options, data)
+
+
class _FakeVectorIndexWriter:
instances = []
- def __init__(self, options):
- self.options = dict(options)
- self.trained = None
+ def __init__(self, training):
+ self.options = dict(training.options)
+ self.trained = training.trained
self.added_ids = None
self.added_vectors = None
self.closed = False
_FakeVectorIndexWriter.instances.append(self)
- def train(self, data):
- self.trained = data.tolist()
-
def add_vectors(self, ids, data):
self.added_ids = ids.tolist()
self.added_vectors = data.tolist()
@@ -551,6 +568,7 @@ class GlobalIndexBuildTest(
old_module = sys.modules.get("paimon_vindex")
sys.modules["paimon_vindex"] = types.SimpleNamespace(
+ VectorIndexTrainer=_FakeVectorIndexTrainer,
VectorIndexWriter=_FakeVectorIndexWriter)
_FakeVectorIndexWriter.instances = []
try:
@@ -609,6 +627,7 @@ class GlobalIndexBuildTest(
old_module = sys.modules.get("paimon_vindex")
sys.modules["paimon_vindex"] = types.SimpleNamespace(
+ VectorIndexTrainer=_FakeVectorIndexTrainer,
VectorIndexWriter=_FakeVectorIndexWriter)
_FakeVectorIndexWriter.instances = []
try:
@@ -632,6 +651,13 @@ class GlobalIndexBuildTest(
[[0, 1], [0, 1], [0]],
[writer.added_ids for writer in _FakeVectorIndexWriter.instances],
)
+ self.assertEqual(
+ ['2', '2', '1'],
+ [
+ writer.options['expected-vector-count']
+ for writer in _FakeVectorIndexWriter.instances
+ ],
+ )
snapshot = table.snapshot_manager().get_latest_snapshot()
entries = sorted(
@@ -669,6 +695,7 @@ class GlobalIndexBuildTest(
old_module = sys.modules.get("paimon_vindex")
sys.modules["paimon_vindex"] = types.SimpleNamespace(
+ VectorIndexTrainer=_FakeVectorIndexTrainer,
VectorIndexWriter=_FakeVectorIndexWriter)
_FakeVectorIndexWriter.instances = []
options = {
@@ -963,6 +990,50 @@ class GlobalIndexBuildTest(
self.assertEqual('16', result['pq.m'])
self.assertEqual('true', result['use-opq'])
+ def test_vindex_native_options_support_030_indexes(self):
+ data_type = ArrayType(True, AtomicType('FLOAT'))
+ options = {
+ 'ivf-rq.rq.bits': '5',
+ 'ivf-rq.max-bytes-per-vector': '96',
+ 'diskann.build-preset': 'balanced',
+ 'diskann.pq.code-ratio': '0.0625',
+ 'diskann.raw-vector-encoding': 'f16',
+ }
+
+ rq_result = native_options(
+ data_type, options, 'ivf-rq', 'embedding')
+ self.assertEqual('ivf_rq', rq_result['index.type'])
+ self.assertEqual('5', rq_result['rq.bits'])
+ self.assertEqual('96', rq_result['max-bytes-per-vector'])
+ self.assertEqual('inner_product', rq_result['metric'])
+
+ diskann_result = native_options(
+ data_type, options, 'diskann', 'embedding')
+ self.assertEqual('diskann', diskann_result['index.type'])
+ self.assertEqual(
+ 'balanced', diskann_result['diskann.build-preset'])
+ self.assertEqual('0.0625', diskann_result['pq.code-ratio'])
+ self.assertEqual(
+ 'f16', diskann_result['diskann.raw-vector-encoding'])
+
+ def test_vindex_training_sample_ratio(self):
+ options = {
+ 'ivf-rq.train.sample-ratio': '0.5',
+ 'fields.embedding.train.sample-ratio': '0.25',
+ }
+ self.assertEqual(
+ 0.25, train_sample_ratio(options, 'ivf-rq', 'embedding'))
+ self.assertEqual(
+ 0.5, train_sample_ratio(options, 'ivf-rq', 'other'))
+
+ import numpy as np
+ vectors = np.arange(20, dtype=np.float32).reshape(10, 2)
+ sampled = _sample_training_vectors(np, vectors, 0.4)
+ self.assertEqual(
+ [[0.0, 1.0], [4.0, 5.0], [10.0, 11.0], [14.0, 15.0]],
+ sampled.tolist(),
+ )
+
def test_split_by_contiguous_row_range_matches_java_builder(self):
split = _FakeSplit([
_FakeFile('a', 0, 2),
diff --git a/paimon-python/pypaimon/tests/vindex_vector_index_test.py
b/paimon-python/pypaimon/tests/vindex_vector_index_test.py
index a549c43af8..76715d253f 100644
--- a/paimon-python/pypaimon/tests/vindex_vector_index_test.py
+++ b/paimon-python/pypaimon/tests/vindex_vector_index_test.py
@@ -36,6 +36,26 @@ class _FakeMetadata:
total_vectors = 100
+class _FakeSearchParams:
+
+ def __init__(self, kind, top_k, width):
+ self.kind = kind
+ self.top_k = top_k
+ self.width = width
+
+ @classmethod
+ def automatic(cls, top_k):
+ return cls("automatic", top_k, 0)
+
+ @classmethod
+ def ivf(cls, top_k, nprobe):
+ return cls("ivf", top_k, nprobe)
+
+ @classmethod
+ def diskann(cls, top_k, l_search):
+ return cls("diskann", top_k, l_search)
+
+
class _FakeVectorIndexReader:
instances = []
@@ -49,20 +69,22 @@ class _FakeVectorIndexReader:
def metadata(self):
return _FakeMetadata()
- def search(self, query, effective_k, nprobe, ef_search, filter_bytes=None):
+ def optimize_for_search(self):
+ pass
+
+ def search(self, query, params, filter_bytes=None):
self.search_calls.append(
- (list(query), effective_k, nprobe, ef_search, filter_bytes))
- return list(range(10, 10 + effective_k)), [float(i) for i in
range(effective_k)]
+ (list(query), params, filter_bytes))
+ return (
+ list(range(10, 10 + params.top_k)),
+ [float(i) for i in range(params.top_k)],
+ )
- def search_batch(
- self, queries, top_k, nprobe, ef_search=0, filter_bytes=None
- ):
+ def search_batch(self, queries, params, filter_bytes=None):
self.batch_calls.append(
{
"queries": queries,
- "top_k": top_k,
- "nprobe": nprobe,
- "ef_search": ef_search,
+ "params": params,
"filter_bytes": filter_bytes,
}
)
@@ -71,7 +93,7 @@ class _FakeVectorIndexReader:
query_count = queries.shape[0]
for query_index in range(query_count):
base_id = (query_index + 1) * 10
- for rank in range(top_k):
+ for rank in range(params.top_k):
ids.append(base_id + rank)
distances.append(float(rank))
return ids, distances
@@ -111,6 +133,7 @@ class VindexVectorIndexTest(unittest.TestCase):
def test_batch_search_uses_native_batch_api(self):
old_module = sys.modules.get("paimon_vindex")
sys.modules["paimon_vindex"] = types.SimpleNamespace(
+ SearchParams=_FakeSearchParams,
VectorIndexReader=_FakeVectorIndexReader)
_FakeVectorIndexReader.instances = []
@@ -130,7 +153,7 @@ class VindexVectorIndexTest(unittest.TestCase):
vectors=[[1.0, 0.0], [0.0, 1.0], [0.5, 0.5]],
limit=3,
field_name="embedding",
- options={"ivf.nprobe": "7", "hnsw.ef_search": "11"},
+ options={"ivf.nprobe": "7"},
).with_include_row_ids(include_ids)
).result()
@@ -140,9 +163,9 @@ class VindexVectorIndexTest(unittest.TestCase):
call = fake_reader.batch_calls[0]
self.assertEqual((3, 2), call["queries"].shape)
- self.assertEqual(2, call["top_k"])
- self.assertEqual(7, call["nprobe"])
- self.assertEqual(11, call["ef_search"])
+ self.assertEqual(2, call["params"].top_k)
+ self.assertEqual("ivf", call["params"].kind)
+ self.assertEqual(7, call["params"].width)
self.assertIsNotNone(call["filter_bytes"])
self.assertEqual(3, len(results))
diff --git a/paimon-python/setup.py b/paimon-python/setup.py
index f6065b4ab7..85c4efcd56 100644
--- a/paimon-python/setup.py
+++ b/paimon-python/setup.py
@@ -181,7 +181,7 @@ setup(
'lumina-data>=0.1.0'
],
'vindex': [
- 'paimon-vindex==0.1.0; python_version>="3.9"',
+ 'paimon-vindex==0.3.0; python_version>="3.9"',
],
'full-text': [
'paimon-ftindex==0.1.0; python_version>="3.8"',
diff --git a/paimon-vector/pom.xml b/paimon-vector/pom.xml
index fbe31c2fd5..dc422b7cd8 100644
--- a/paimon-vector/pom.xml
+++ b/paimon-vector/pom.xml
@@ -32,7 +32,7 @@ under the License.
<name>Paimon : Vector Index</name>
<properties>
-
<paimon-vector-index-java.version>0.2.0</paimon-vector-index-java.version>
+
<paimon-vector-index-java.version>0.3.0</paimon-vector-index-java.version>
</properties>
<dependencies>
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswSqVectorGlobalIndexerFactory.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/DiskAnnVectorGlobalIndexerFactory.java
similarity index 80%
copy from
paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswSqVectorGlobalIndexerFactory.java
copy to
paimon-vector/src/main/java/org/apache/paimon/vector/index/DiskAnnVectorGlobalIndexerFactory.java
index 3f09984db5..f8b5c07e98 100644
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswSqVectorGlobalIndexerFactory.java
+++
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/DiskAnnVectorGlobalIndexerFactory.java
@@ -18,10 +18,10 @@
package org.apache.paimon.vector.index;
-/** Factory for the {@code ivf-hnsw-sq} vector index identifier. */
-public class IvfHnswSqVectorGlobalIndexerFactory extends
NativeVectorGlobalIndexerFactory {
+/** Factory for the {@code diskann} vector index identifier. */
+public class DiskAnnVectorGlobalIndexerFactory extends
NativeVectorGlobalIndexerFactory {
- public static final String IDENTIFIER = "ivf-hnsw-sq";
+ public static final String IDENTIFIER = "diskann";
@Override
public String identifier() {
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswFlatVectorGlobalIndexerFactory.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfRqVectorGlobalIndexerFactory.java
similarity index 80%
rename from
paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswFlatVectorGlobalIndexerFactory.java
rename to
paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfRqVectorGlobalIndexerFactory.java
index 764920d5d7..2b0e60a19b 100644
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswFlatVectorGlobalIndexerFactory.java
+++
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfRqVectorGlobalIndexerFactory.java
@@ -18,10 +18,10 @@
package org.apache.paimon.vector.index;
-/** Factory for the {@code ivf-hnsw-flat} vector index identifier. */
-public class IvfHnswFlatVectorGlobalIndexerFactory extends
NativeVectorGlobalIndexerFactory {
+/** Factory for the {@code ivf-rq} vector index identifier. */
+public class IvfRqVectorGlobalIndexerFactory extends
NativeVectorGlobalIndexerFactory {
- public static final String IDENTIFIER = "ivf-hnsw-flat";
+ public static final String IDENTIFIER = "ivf-rq";
@Override
public String identifier() {
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswSqVectorGlobalIndexerFactory.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfSqVectorGlobalIndexerFactory.java
similarity index 80%
rename from
paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswSqVectorGlobalIndexerFactory.java
rename to
paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfSqVectorGlobalIndexerFactory.java
index 3f09984db5..79dbe95c17 100644
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfHnswSqVectorGlobalIndexerFactory.java
+++
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/IvfSqVectorGlobalIndexerFactory.java
@@ -18,10 +18,10 @@
package org.apache.paimon.vector.index;
-/** Factory for the {@code ivf-hnsw-sq} vector index identifier. */
-public class IvfHnswSqVectorGlobalIndexerFactory extends
NativeVectorGlobalIndexerFactory {
+/** Factory for the {@code ivf-sq} vector index identifier. */
+public class IvfSqVectorGlobalIndexerFactory extends
NativeVectorGlobalIndexerFactory {
- public static final String IDENTIFIER = "ivf-hnsw-sq";
+ public static final String IDENTIFIER = "ivf-sq";
@Override
public String identifier() {
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java
index b525fdd226..4056212e92 100644
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java
+++
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java
@@ -31,6 +31,7 @@ import org.apache.paimon.index.vector.VectorIndexInput;
import org.apache.paimon.index.vector.VectorIndexMetadata;
import org.apache.paimon.index.vector.VectorIndexReader;
import org.apache.paimon.index.vector.VectorSearchBatchResult;
+import org.apache.paimon.index.vector.VectorSearchParams;
import org.apache.paimon.index.vector.VectorSearchResult;
import org.apache.paimon.predicate.BatchVectorSearch;
import org.apache.paimon.predicate.FieldRef;
@@ -63,9 +64,7 @@ import static
org.apache.paimon.utils.Preconditions.checkArgument;
public class NativeVectorGlobalIndexReader implements GlobalIndexReader {
private static final String NPROBE_PARAMETER = "ivf.nprobe";
- private static final String EF_SEARCH_PARAMETER = "hnsw.ef_search";
- private static final int DEFAULT_NPROBE = 16;
- private static final int DEFAULT_EF_SEARCH = 0;
+ private static final String L_SEARCH_PARAMETER = "diskann.l_search";
private static final int VECTOR_INDEX_MIN_SEEK_FOR_VECTOR_READS = 16 *
1024;
private static final int VECTOR_INDEX_PARALLELISM_FOR_VECTOR_READS = 32;
@@ -145,8 +144,6 @@ public class NativeVectorGlobalIndexReader implements
GlobalIndexReader {
validateSearchVector(vector);
}
int dim = nativeMeta.dimension();
- int nprobe = nprobe(batchVectorSearch.options());
- int efSearch = efSearch(batchVectorSearch.options());
String metric = nativeMeta.metric();
SearchScope scope =
@@ -164,8 +161,14 @@ public class NativeVectorGlobalIndexReader implements
GlobalIndexReader {
VectorSearchBatchResult batchResult =
scope.filterBytes != null
? vectorReader.searchBatch(
- queries, n, scope.effectiveK, nprobe,
efSearch, scope.filterBytes)
- : vectorReader.searchBatch(queries, n,
scope.effectiveK, nprobe, efSearch);
+ queries,
+ n,
+ searchParams(batchVectorSearch.options(),
scope.effectiveK),
+ scope.filterBytes)
+ : vectorReader.searchBatch(
+ queries,
+ n,
+ searchParams(batchVectorSearch.options(),
scope.effectiveK));
// result i corresponds to vectors[i], matching input order.
List<Optional<ScoredGlobalIndexResult>> results = new ArrayList<>(n);
@@ -181,8 +184,6 @@ public class NativeVectorGlobalIndexReader implements
GlobalIndexReader {
validateSearchVector(vectorSearch.vector());
float[] queryVector = vectorSearch.vector().clone();
int limit = vectorSearch.limit();
- int nprobe = nprobe(vectorSearch.options());
- int efSearch = efSearch(vectorSearch.options());
String metric = nativeMeta.metric();
SearchScope scope = resolveScope(vectorSearch.includeRowIds(), limit);
@@ -192,8 +193,12 @@ public class NativeVectorGlobalIndexReader implements
GlobalIndexReader {
VectorSearchResult result =
scope.filterBytes != null
? vectorReader.search(
- queryVector, scope.effectiveK, nprobe,
efSearch, scope.filterBytes)
- : vectorReader.search(queryVector, scope.effectiveK,
nprobe, efSearch);
+ queryVector,
+ searchParams(vectorSearch.options(),
scope.effectiveK),
+ scope.filterBytes)
+ : vectorReader.search(
+ queryVector,
+ searchParams(vectorSearch.options(),
scope.effectiveK));
return buildScoredResult(result.ids(), result.distances(),
metric).orElse(null);
}
@@ -279,18 +284,26 @@ public class NativeVectorGlobalIndexReader implements
GlobalIndexReader {
throw new IllegalArgumentException("Unknown metric: " + metric);
}
- static int nprobe(Map<String, String> parameters) {
- return intParameter(parameters, NPROBE_PARAMETER, DEFAULT_NPROBE);
- }
-
- static int efSearch(Map<String, String> parameters) {
- return intParameter(parameters, EF_SEARCH_PARAMETER,
DEFAULT_EF_SEARCH);
+ static VectorSearchParams searchParams(Map<String, String> parameters, int
topK) {
+ Integer nprobe = intParameter(parameters, NPROBE_PARAMETER);
+ Integer lSearch = intParameter(parameters, L_SEARCH_PARAMETER);
+ if (nprobe != null && lSearch != null) {
+ throw new IllegalArgumentException(
+ "Cannot set both '" + NPROBE_PARAMETER + "' and '" +
L_SEARCH_PARAMETER + "'.");
+ }
+ if (nprobe != null) {
+ return VectorSearchParams.ivf(topK, nprobe);
+ }
+ if (lSearch != null) {
+ return VectorSearchParams.diskAnn(topK, lSearch);
+ }
+ return VectorSearchParams.automatic(topK);
}
- private static int intParameter(Map<String, String> parameters, String
key, int defaultValue) {
+ private static Integer intParameter(Map<String, String> parameters, String
key) {
String value = parameters.get(key);
if (value == null) {
- return defaultValue;
+ return null;
}
try {
return Integer.parseInt(value);
@@ -330,14 +343,21 @@ public class NativeVectorGlobalIndexReader implements
GlobalIndexReader {
synchronized (this) {
if (vectorReader == null) {
SeekableInputStream in = fileReader.getInputStream(ioMeta);
+ VectorIndexReader reader = null;
try {
- NativeVectorIndexLoader.loadJni();
- VectorIndexReader reader =
- new VectorIndexReader(new
SeekableStreamVectorIndexInput(in));
+ reader = new VectorIndexReader(new
SeekableStreamVectorIndexInput(in));
nativeMeta = reader.metadata();
+ reader.optimizeForSearch();
vectorReader = reader;
openStream = in;
} catch (Exception e) {
+ if (reader != null) {
+ try {
+ reader.close();
+ } catch (Exception closeException) {
+ e.addSuppressed(closeException);
+ }
+ }
IOUtils.closeQuietly(in);
throw e;
}
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java
index 6ff6843f00..d0f52288de 100644
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java
+++
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java
@@ -44,6 +44,7 @@ import java.nio.ByteOrder;
import java.nio.channels.FileChannel;
import java.util.Arrays;
import java.util.Collections;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -252,7 +253,6 @@ public class NativeVectorGlobalIndexWriter implements
GlobalIndexSingleColumnWri
LOG.info("{} vector index build started: {} vectors, dim={}",
identifier, count, dim);
long buildStart = System.currentTimeMillis();
- NativeVectorIndexLoader.loadJni();
// Phase 1: Train
long phaseStart = System.currentTimeMillis();
LOG.info("{} train phase started", identifier);
@@ -305,7 +305,7 @@ public class NativeVectorGlobalIndexWriter implements
GlobalIndexSingleColumnWri
float[] batchVectors = new float[trainBatchSize * dim];
logTrainingMemoryEstimate(trainCount);
- try (VectorIndexTrainer trainer =
VectorIndexTrainer.create(nativeOptions);
+ try (VectorIndexTrainer trainer =
VectorIndexTrainer.create(trainingOptions());
RandomAccessFile raf = new RandomAccessFile(tempVectorFile,
"r");
FileChannel channel = raf.getChannel()) {
ByteBuffer readBuf = ByteBuffer.allocateDirect(IO_BUFFER_SIZE);
@@ -355,6 +355,19 @@ public class NativeVectorGlobalIndexWriter implements
GlobalIndexSingleColumnWri
}
}
+ private Map<String, String> trainingOptions() {
+ Map<String, String> options = new LinkedHashMap<>(nativeOptions);
+ String indexType = options.get("index.type");
+ String nlist = options.get("nlist");
+ if (indexType != null
+ && indexType.startsWith("ivf_")
+ && (nlist == null || "auto".equals(nlist))
+ && !options.containsKey("expected-vector-count")) {
+ options.put("expected-vector-count", Long.toString(count));
+ }
+ return options;
+ }
+
private void addVectorsFromTempFile(VectorIndexWriter writer) throws
IOException {
int addBatchSize = vectorBatchSize(ADD_BATCH_SIZE, dim);
long[] batchIds = new long[addBatchSize];
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java
index 3293d1620e..37e20e0af5 100644
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java
+++
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java
@@ -34,7 +34,7 @@ import java.util.concurrent.ExecutorService;
/** Native vector global indexer backed by paimon-vector-index-java. */
public class NativeVectorGlobalIndexer implements VectorGlobalIndexer {
- private static final String DEFAULT_METRIC = "inner_product";
+ static final String DEFAULT_METRIC = "inner_product";
private final DataType fieldType;
private final Map<String, String> options;
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java
index c351697518..0f28479fa0 100644
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java
+++
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java
@@ -78,6 +78,7 @@ public abstract class NativeVectorGlobalIndexerFactory
implements GlobalIndexerF
nativeOptions.put("index.type", identifier.replace('-', '_'));
nativeOptions.put(
"dimension", String.valueOf(dimension(fieldType,
nativeOptions, identifier)));
+ nativeOptions.putIfAbsent("metric",
NativeVectorGlobalIndexer.DEFAULT_METRIC);
return nativeOptions;
}
@@ -146,14 +147,45 @@ public abstract class NativeVectorGlobalIndexerFactory
implements GlobalIndexerF
case "metric":
return "metric";
case "nlist":
+ case "expected-vector-count":
case "pq.m":
- case "hnsw.m":
- case "hnsw.ef-construction":
- case "hnsw.max-level":
+ case "pq.code-ratio":
+ case "pq.bits":
+ case "rq.bits":
+ case "target-recall":
+ case "max-bytes-per-vector":
+ case "deployment-profile":
return optionKey;
case "pq.use-opq":
case "use-opq":
return "use-opq";
+ case "build-preset":
+ case "diskann.build-preset":
+ return "diskann.build-preset";
+ case "max-degree":
+ case "diskann.max-degree":
+ return "diskann.max-degree";
+ case "build-search-list-size":
+ case "diskann.build-search-list-size":
+ return "diskann.build-search-list-size";
+ case "alpha":
+ case "diskann.alpha":
+ return "diskann.alpha";
+ case "seed":
+ case "diskann.seed":
+ return "diskann.seed";
+ case "memory-budget-bytes":
+ case "diskann.memory-budget-bytes":
+ return "diskann.memory-budget-bytes";
+ case "storage-layout":
+ case "diskann.storage-layout":
+ return "diskann.storage-layout";
+ case "raw-vector-encoding":
+ case "diskann.raw-vector-encoding":
+ return "diskann.raw-vector-encoding";
+ case "build-distance":
+ case "diskann.build-distance":
+ return "diskann.build-distance";
default:
return null;
}
diff --git
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorIndexLoader.java
b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorIndexLoader.java
deleted file mode 100644
index 3c55fda25f..0000000000
---
a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorIndexLoader.java
+++ /dev/null
@@ -1,112 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.paimon.vector.index;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.util.Locale;
-
-/** Loads native libraries packaged by paimon-vector-index-java. */
-final class NativeVectorIndexLoader {
-
- private static volatile boolean loaded;
-
- private NativeVectorIndexLoader() {}
-
- static void loadJni() {
- if (loaded) {
- return;
- }
- synchronized (NativeVectorIndexLoader.class) {
- if (loaded) {
- return;
- }
- loadFromResource();
- loaded = true;
- }
- }
-
- private static void loadFromResource() {
- String libraryPath = nativeLibraryPath();
- try (InputStream input =
NativeVectorIndexLoader.class.getResourceAsStream(libraryPath)) {
- if (input == null) {
- throw new UnsupportedOperationException(
- "Vector index native library resource not found: " +
libraryPath);
- }
-
- File temp = File.createTempFile("paimon-vector-", librarySuffix());
- temp.deleteOnExit();
- try (FileOutputStream output = new FileOutputStream(temp)) {
- byte[] buffer = new byte[8192];
- int len;
- while ((len = input.read(buffer)) >= 0) {
- output.write(buffer, 0, len);
- }
- }
- System.load(temp.getAbsolutePath());
- } catch (IOException e) {
- throw new RuntimeException(
- "Failed to load vector index native library: " +
libraryPath, e);
- }
- }
-
- private static String nativeLibraryPath() {
- return "/native/" + os() + "/" + arch() + "/" + libraryName();
- }
-
- private static String os() {
- String os = System.getProperty("os.name").toLowerCase(Locale.ROOT);
- if (os.contains("mac")) {
- return "macos";
- } else if (os.contains("linux")) {
- return "linux";
- } else if (os.contains("windows")) {
- return "windows";
- }
- throw new UnsupportedOperationException("Unsupported vector index
native OS: " + os);
- }
-
- private static String arch() {
- String arch = System.getProperty("os.arch").toLowerCase(Locale.ROOT);
- if ("amd64".equals(arch) || "x86_64".equals(arch)) {
- return "x86_64";
- } else if ("aarch64".equals(arch) || "arm64".equals(arch)) {
- return "aarch64";
- }
- throw new UnsupportedOperationException("Unsupported vector index
native arch: " + arch);
- }
-
- private static String libraryName() {
- if ("windows".equals(os())) {
- return "paimon_vindex_jni.dll";
- }
- return "libpaimon_vindex_jni" + librarySuffix();
- }
-
- private static String librarySuffix() {
- if ("macos".equals(os())) {
- return ".dylib";
- } else if ("windows".equals(os())) {
- return ".dll";
- }
- return ".so";
- }
-}
diff --git
a/paimon-vector/src/main/resources/META-INF/services/org.apache.paimon.globalindex.GlobalIndexerFactory
b/paimon-vector/src/main/resources/META-INF/services/org.apache.paimon.globalindex.GlobalIndexerFactory
index a11570704a..28a1439e57 100644
---
a/paimon-vector/src/main/resources/META-INF/services/org.apache.paimon.globalindex.GlobalIndexerFactory
+++
b/paimon-vector/src/main/resources/META-INF/services/org.apache.paimon.globalindex.GlobalIndexerFactory
@@ -15,5 +15,6 @@
org.apache.paimon.vector.index.IvfFlatVectorGlobalIndexerFactory
org.apache.paimon.vector.index.IvfPqAlgorithmVectorGlobalIndexerFactory
-org.apache.paimon.vector.index.IvfHnswFlatVectorGlobalIndexerFactory
-org.apache.paimon.vector.index.IvfHnswSqVectorGlobalIndexerFactory
+org.apache.paimon.vector.index.IvfSqVectorGlobalIndexerFactory
+org.apache.paimon.vector.index.IvfRqVectorGlobalIndexerFactory
+org.apache.paimon.vector.index.DiskAnnVectorGlobalIndexerFactory
diff --git
a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java
b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java
index 82b345b799..4995f3e886 100644
---
a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java
+++
b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java
@@ -27,6 +27,7 @@ import org.apache.paimon.globalindex.ResultEntry;
import org.apache.paimon.globalindex.ScoredGlobalIndexResult;
import org.apache.paimon.globalindex.io.GlobalIndexFileReader;
import org.apache.paimon.globalindex.io.GlobalIndexFileWriter;
+import org.apache.paimon.index.vector.VectorSearchParams;
import org.apache.paimon.options.Options;
import org.apache.paimon.predicate.BatchVectorSearch;
import org.apache.paimon.predicate.VectorSearch;
@@ -72,7 +73,6 @@ public class NativeVectorGlobalIndexTest {
private static boolean isNativeAvailable() {
try {
- NativeVectorIndexLoader.loadJni();
Options options = new Options();
options.setInteger("ivf-flat.dimension", 2);
options.setString("ivf-flat.metric", "l2");
@@ -240,25 +240,38 @@ public class NativeVectorGlobalIndexTest {
public void testVectorSearchParameterParsing() {
Map<String, String> parameters = new HashMap<>();
parameters.put("ivf.nprobe", "24");
- parameters.put("hnsw.ef_search", "80");
parameters.put("ignored", "bad");
-
assertThat(NativeVectorGlobalIndexReader.nprobe(parameters)).isEqualTo(24);
-
assertThat(NativeVectorGlobalIndexReader.efSearch(parameters)).isEqualTo(80);
-
assertThat(NativeVectorGlobalIndexReader.nprobe(Collections.emptyMap())).isEqualTo(16);
-
assertThat(NativeVectorGlobalIndexReader.efSearch(Collections.emptyMap())).isEqualTo(0);
+ VectorSearchParams ivfParams =
NativeVectorGlobalIndexReader.searchParams(parameters, 10);
+ assertThat(ivfParams.topK()).isEqualTo(10);
+
+ VectorSearchParams automaticParams =
+
NativeVectorGlobalIndexReader.searchParams(Collections.emptyMap(), 5);
+ assertThat(automaticParams.topK()).isEqualTo(5);
+
+ VectorSearchParams diskAnnParams =
+ NativeVectorGlobalIndexReader.searchParams(
+ Collections.singletonMap("diskann.l_search", "80"),
10);
+ assertThat(diskAnnParams.topK()).isEqualTo(10);
}
@Test
public void testVectorSearchParameterRangeValidationDelegatedToNative() {
assertThat(
- NativeVectorGlobalIndexReader.nprobe(
- Collections.singletonMap("ivf.nprobe", "0")))
- .isEqualTo(0);
+ NativeVectorGlobalIndexReader.searchParams(
+ Collections.singletonMap("ivf.nprobe", "0"),
10))
+ .isNotNull();
assertThat(
- NativeVectorGlobalIndexReader.efSearch(
- Collections.singletonMap("hnsw.ef_search",
"-1")))
- .isEqualTo(-1);
+ NativeVectorGlobalIndexReader.searchParams(
+ Collections.singletonMap("diskann.l_search",
"-1"), 10))
+ .isNotNull();
+
+ Map<String, String> conflicting = new HashMap<>();
+ conflicting.put("ivf.nprobe", "16");
+ conflicting.put("diskann.l_search", "100");
+ assertThatThrownBy(() ->
NativeVectorGlobalIndexReader.searchParams(conflicting, 10))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("Cannot set both");
}
@Test
@@ -320,6 +333,57 @@ public class NativeVectorGlobalIndexTest {
}
}
+ @Test
+ public void testNewIndexTypesEndToEnd() throws IOException {
+ Assumptions.assumeTrue(isNativeAvailable(), "Vector index native
library not available");
+
+ float[][] vectors = new float[32][2];
+ for (int i = 0; i < vectors.length; i++) {
+ double angle = 2 * Math.PI * i / vectors.length;
+ vectors[i][0] = (float) Math.cos(angle);
+ vectors[i][1] = (float) Math.sin(angle);
+ }
+
+ for (String identifier :
+ new String[] {
+ IvfSqVectorGlobalIndexerFactory.IDENTIFIER,
+ IvfRqVectorGlobalIndexerFactory.IDENTIFIER,
+ DiskAnnVectorGlobalIndexerFactory.IDENTIFIER
+ }) {
+ Options options = new Options();
+ options.setInteger(identifier + ".dimension", 2);
+ options.setString(identifier + ".metric", "l2");
+ if (IvfRqVectorGlobalIndexerFactory.IDENTIFIER.equals(identifier))
{
+ options.setInteger(identifier + ".nlist", 1);
+ } else {
+ if
(DiskAnnVectorGlobalIndexerFactory.IDENTIFIER.equals(identifier)) {
+ options.setInteger("diskann.max-degree", 4);
+ options.setInteger("diskann.build-search-list-size", 8);
+ }
+ }
+
+ NativeVectorGlobalIndexWriter writer =
+ new NativeVectorGlobalIndexWriter(
+ createFileWriter(indexPath),
+ vectorType,
+ NativeVectorGlobalIndexerFactory.nativeOptions(
+ vectorType, options, identifier,
fieldName),
+ identifier);
+ writeVectors(writer, vectors);
+ List<GlobalIndexIOMeta> metas = toIOMetas(writer.finish(),
indexPath);
+
+ try (NativeVectorGlobalIndexReader reader =
+ new NativeVectorGlobalIndexReader(
+ createFileReader(indexPath), metas, vectorType,
executor)) {
+ ScoredGlobalIndexResult result =
+ reader.visitVectorSearch(new VectorSearch(vectors[0],
2, fieldName))
+ .join()
+ .get();
+
assertThat(result.results().contains(0L)).as(identifier).isTrue();
+ }
+ }
+ }
+
@Test
public void testSearchWithRoaringFilter() throws IOException {
Assumptions.assumeTrue(isNativeAvailable(), "Vector index native
library not available");
diff --git
a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java
b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java
index 856a38185f..6c30913491 100644
---
a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java
+++
b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java
@@ -38,9 +38,9 @@ public class NativeVectorGlobalIndexerFactoryTest {
public void testIdentifier() {
assertThat(new
IvfFlatVectorGlobalIndexerFactory().identifier()).isEqualTo("ivf-flat");
assertThat(new
IvfPqAlgorithmVectorGlobalIndexerFactory().identifier()).isEqualTo("ivf-pq");
- assertThat(new IvfHnswFlatVectorGlobalIndexerFactory().identifier())
- .isEqualTo("ivf-hnsw-flat");
- assertThat(new
IvfHnswSqVectorGlobalIndexerFactory().identifier()).isEqualTo("ivf-hnsw-sq");
+ assertThat(new
IvfSqVectorGlobalIndexerFactory().identifier()).isEqualTo("ivf-sq");
+ assertThat(new
IvfRqVectorGlobalIndexerFactory().identifier()).isEqualTo("ivf-rq");
+ assertThat(new
DiskAnnVectorGlobalIndexerFactory().identifier()).isEqualTo("diskann");
}
@Test
@@ -49,10 +49,12 @@ public class NativeVectorGlobalIndexerFactoryTest {
.isExactlyInstanceOf(IvfFlatVectorGlobalIndexerFactory.class);
assertThat(GlobalIndexerFactoryUtils.load("ivf-pq"))
.isExactlyInstanceOf(IvfPqAlgorithmVectorGlobalIndexerFactory.class);
- assertThat(GlobalIndexerFactoryUtils.load("ivf-hnsw-flat"))
-
.isExactlyInstanceOf(IvfHnswFlatVectorGlobalIndexerFactory.class);
- assertThat(GlobalIndexerFactoryUtils.load("ivf-hnsw-sq"))
-
.isExactlyInstanceOf(IvfHnswSqVectorGlobalIndexerFactory.class);
+ assertThat(GlobalIndexerFactoryUtils.load("ivf-sq"))
+ .isExactlyInstanceOf(IvfSqVectorGlobalIndexerFactory.class);
+ assertThat(GlobalIndexerFactoryUtils.load("ivf-rq"))
+ .isExactlyInstanceOf(IvfRqVectorGlobalIndexerFactory.class);
+ assertThat(GlobalIndexerFactoryUtils.load("diskann"))
+ .isExactlyInstanceOf(DiskAnnVectorGlobalIndexerFactory.class);
}
@Test
@@ -84,6 +86,51 @@ public class NativeVectorGlobalIndexerFactoryTest {
.doesNotContainKey("vector.file.format");
}
+ @Test
+ public void testNativeOptionsUseDefaultMetric() {
+ Map<String, String> nativeOptions =
+ NativeVectorGlobalIndexerFactory.nativeOptions(
+ new ArrayType(new FloatType()),
+ new Options(),
+ IvfFlatVectorGlobalIndexerFactory.IDENTIFIER,
+ "vec");
+
+ assertThat(nativeOptions).containsEntry("metric", "inner_product");
+ }
+
+ @Test
+ public void testNewVectorIndexOptions() {
+ Options options = new Options();
+ options.setString("ivf-rq.rq.bits", "5");
+ options.setString("ivf-rq.max-bytes-per-vector", "96");
+ options.setString("diskann.build-preset", "balanced");
+ options.setString("diskann.pq.code-ratio", "0.0625");
+ options.setString("diskann.raw-vector-encoding", "f16");
+
+ Map<String, String> rqOptions =
+ NativeVectorGlobalIndexerFactory.nativeOptions(
+ new ArrayType(new FloatType()),
+ options,
+ IvfRqVectorGlobalIndexerFactory.IDENTIFIER,
+ "vec");
+ assertThat(rqOptions)
+ .containsEntry("index.type", "ivf_rq")
+ .containsEntry("rq.bits", "5")
+ .containsEntry("max-bytes-per-vector", "96");
+
+ Map<String, String> diskAnnOptions =
+ NativeVectorGlobalIndexerFactory.nativeOptions(
+ new ArrayType(new FloatType()),
+ options,
+ DiskAnnVectorGlobalIndexerFactory.IDENTIFIER,
+ "vec");
+ assertThat(diskAnnOptions)
+ .containsEntry("index.type", "diskann")
+ .containsEntry("diskann.build-preset", "balanced")
+ .containsEntry("pq.code-ratio", "0.0625")
+ .containsEntry("diskann.raw-vector-encoding", "f16");
+ }
+
@Test
public void testNativeOptionsUsesVectorTypeDimension() {
Options options = new Options();
