dungba88 commented on code in PR #14708:
URL: https://github.com/apache/lucene/pull/14708#discussion_r2115082489
##########
lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityValuesSource.java:
##########
@@ -42,7 +78,35 @@ public VectorScorer getScorer(LeafReaderContext ctx) throws
IOException {
ByteVectorValues.checkField(ctx.reader(), fieldName);
return null;
}
- return vectorValues.scorer(queryVector);
+ final FieldInfo fi = ctx.reader().getFieldInfos().fieldInfo(fieldName);
+ if (fi.getVectorDimension() != queryVector.length) {
+ throw new IllegalArgumentException(
+ "Query vector dimension does not match field dimension: "
+ + queryVector.length
+ + " != "
+ + fi.getVectorDimension());
+ }
+
+ // default vector scorer
+ if (useFullPrecision == false) {
+ return vectorValues.scorer(queryVector);
+ }
+
+ final VectorSimilarityFunction vectorSimilarityFunction =
fi.getVectorSimilarityFunction();
+ return new VectorScorer() {
+ final KnnVectorValues.DocIndexIterator iterator =
vectorValues.iterator();
+
+ @Override
+ public float score() throws IOException {
+ return vectorSimilarityFunction.compare(
+ queryVector, vectorValues.vectorValue(iterator.index()));
Review Comment:
One of the possible concern for the reranking is that calling this will page
in the full vector data (or what is being used in the iterator), and can
compete the RAM with normal vectors used in HNSW graph search. As HNSW will
suffer the performance if the vectors are not in RAM, I'm wondering if we can
restrict the memory used by the re-ranking phase. Or alternatively, using
`mlock` (could be tricky in Java) to lock in the normal vectors page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
