LantaoJin opened a new pull request, #16829:
URL: https://github.com/apache/lucene/pull/16829
### Description
`Lucene99MemorySegmentScalarQuantizedVectorScorer`, the off-heap scorer for
the legacy Lucene99 scalar-quantized format added in #14863, cannot score a
vector whose bytes straddle an `MMapDirectory` chunk boundary. `getNode` copies
such a node into a heap `scratch` array and wraps it with
`MemorySegment.ofArray`, then calls `node.reinterpret(vectorByteSize)`.
`reinterpret` is defined only for native segments, so every search or graph
build that reaches a straddling node throws `UnsupportedOperationException: Not
a native segment`.
Replacing `reinterpret` with `asSlice` exposes a second problem.
`UpdateableRandomVectorScorerImpl.setScoringOrdinal` keeps the query as a view
of that same `scratch` array, so the next straddling node that is scored
overwrites the query, and the pair scores as the node against itself (1.0 for
`EUCLIDEAN`). During a merge-time HNSW build that would silently degrade the
graph instead of failing.
This PR:
- uses `node.asSlice(0, vectorByteSize)` instead of `reinterpret`, and
removes the now unneeded `@SuppressWarnings("restricted")`;
- copies the query node into its own array in `setScoringOrdinal` when
`getNode` returned a heap copy. Nodes that do not straddle stay off-heap, and
the native path already accepts heap segments (`Linker.Option.critical(true)`).
With the default 16 GiB chunk size this needs a single quantized vectors
file larger than 16 GiB, for example a force-merged segment with about 22M
768-dimension 7-bit vectors. Lucene no longer writes this format, but it still
reads it, and downstream projects that kept a copy of the legacy writer
(OpenSearch k-NN does so for 7-bit quantization) also hit it in merges. The bug
ships in 10.4 and 10.5; the same change applies to the `java21` copy of the
class on `branch_10x`.
**Testing.** `TestVectorScorer` had small-chunk tests for the byte and float
scorers but not for this one. The new `testQuantizedScorerSmallChunkSize`
(every node straddles) and `testQuantizedScorerMedChunkSize` (some do) compare
it with `Lucene99ScalarQuantizedVectorScorer` for 4 and 7 bits, all four
similarities, and both the supplier and the query scorer:
| Code | Result |
|---|---|
| main | both tests fail with `UnsupportedOperationException: Not a native
segment` |
| `asSlice` only | both fail, e.g. expected 0.6477 but was 1.2966 (query
overwritten) |
| this PR | pass, also with `-Ptests.iters=30` |
`./gradlew :lucene:core:check -x test`, the core vectorization tests, and
the backward-codecs Lucene99 quantized format tests with
`-Ptests.defaultvectorization=true` also pass. I did not run the native path
(`-Dlucene.useNativeDotProduct=true`), because no native library was available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
