mccullocht commented on code in PR #16030:
URL: https://github.com/apache/lucene/pull/16030#discussion_r4049355969
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsWriter.java:
##########
@@ -456,8 +662,13 @@ static int mergeAndRecalculateCentroids(
float[] centroid = getCentroid(knnVectorsReader, fieldInfo.name);
totalVectorCount += vectorCount;
// If there aren't centroids, or previously clustered with more than one
cluster
- // or if there are deleted docs, we must recalculate the centroid
- if (centroid == null || mergeState.liveDocs[i] != null) {
+ // or if there are deleted docs, we must recalculate the centroid. A
data-blind segment
+ // stores no centroid (its vectors were quantized against zero); it
can't be combined with
+ // the others, so recompute from the (possibly dequantized) vectors.
Review Comment:
Fixed
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsWriter.java:
##########
@@ -116,13 +127,26 @@ public Lucene104ScalarQuantizedVectorsWriter(
@Override
public FlatFieldVectorsWriter<?> addField(FieldInfo fieldInfo) throws
IOException {
- FlatFieldVectorsWriter<?> rawVectorDelegate =
this.rawVectorDelegate.addField(fieldInfo);
+ if (fieldInfo.getVectorEncoding().isFloatingPoint() && mode ==
Mode.DATA_BLIND_WITHOUT_FLOATS) {
+ // Data-blind mode without floats: keep vectors in memory only and never
write
+ // full-precision float vectors.
+ FlatFieldVectorsWriter<?> storage =
+ switch (fieldInfo.getVectorEncoding()) {
+ case FLOAT32 -> new InMemoryFieldWriter<float[]>(fieldInfo,
Float.BYTES);
+ case FLOAT16 -> new InMemoryFieldWriter<short[]>(fieldInfo,
Short.BYTES);
+ case BYTE -> throw new UnsupportedOperationException("Byte Vectors
aren't supported");
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
