mayya-sharipova commented on code in PR #13090:
URL: https://github.com/apache/lucene/pull/13090#discussion_r1483044238
##########
lucene/core/src/java/org/apache/lucene/util/ScalarQuantizer.java:
##########
@@ -281,24 +270,57 @@ static ScalarQuantizer fromVectors(
}
return new ScalarQuantizer(min, max, confidenceInterval);
}
- int dim = floatVectorValues.dimension();
+ final float[] quantileGatheringScratch =
+ new float[floatVectorValues.dimension() * Math.min(SCRATCH_SIZE,
totalVectorCount)];
+ int count = 0;
+ double upperSum = 0;
+ double lowerSum = 0;
if (totalVectorCount <= quantizationSampleSize) {
- int copyOffset = 0;
- float[] values = new float[totalVectorCount * dim];
+ int scratchSize = Math.min(SCRATCH_SIZE, totalVectorCount);
+ int i = 0;
while (floatVectorValues.nextDoc() != NO_MORE_DOCS) {
- float[] floatVector = floatVectorValues.vectorValue();
- System.arraycopy(floatVector, 0, values, copyOffset,
floatVector.length);
- copyOffset += dim;
+ float[] vectorValue = floatVectorValues.vectorValue();
+ System.arraycopy(
+ vectorValue, 0, quantileGatheringScratch, i * vectorValue.length,
vectorValue.length);
+ i++;
+ if (i == scratchSize) {
Review Comment:
Are we are ok to discard the last left-over (< 20) vectors for calculations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
