Pulkitg64 opened a new issue, #16762: URL: https://github.com/apache/lucene/issues/16762
### Description `Lucene104ScalarQuantizedVectorsWriter` computes a per-field centroid before quantizing the full-precision vectors during a merge. The centroid is the mean of the full-precision vector values gathered from the segments being merged, and it is later used as the reference point for scalar quantization of every vector in the merged segment. When any source segment has deletions, the recalculation path does not skip the deleted documents. ([Code Ref](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsWriter.java#L636-L674)) ``` static int calculateCentroid(MergeState mergeState, FieldInfo fieldInfo, float[] centroid) throws IOException { assert fieldInfo.getVectorEncoding().isFloatingPoint(); // clear out the centroid Arrays.fill(centroid, 0); int count = 0; for (int i = 0; i < mergeState.knnVectorsReaders.length; i++) { KnnVectorsReader knnVectorsReader = mergeState.knnVectorsReaders[i]; if (knnVectorsReader == null) continue; count += accumulateCentroid(knnVectorsReader, fieldInfo, centroid); } if (count == 0) { return count; } for (int i = 0; i < centroid.length; i++) { centroid[i] /= count; } if (fieldInfo.getVectorSimilarityFunction() == COSINE) { VectorUtil.l2normalize(centroid); } return count; } private static int accumulateCentroid( KnnVectorsReader reader, FieldInfo fieldInfo, float[] centroid) throws IOException { FloatVectorValues vectorValues = floatingPointVectorValues(reader, fieldInfo); if (vectorValues == null) { return 0; } int count = 0; KnnVectorValues.DocIndexIterator iterator = vectorValues.iterator(); for (int doc = iterator.nextDoc(); doc != NO_MORE_DOCS; doc = iterator.nextDoc()) { count++; float[] vector = vectorValues.vectorValue(iterator.index()); for (int j = 0; j < vector.length; j++) { centroid[j] += vector[j]; } } return count; } ``` ### Version and environment details _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
