benwtrent commented on code in PR #13090:
URL: https://github.com/apache/lucene/pull/13090#discussion_r1483066398
##########
lucene/core/src/java/org/apache/lucene/util/ScalarQuantizer.java:
##########
@@ -281,24 +270,57 @@ static ScalarQuantizer fromVectors(
}
return new ScalarQuantizer(min, max, confidenceInterval);
}
- int dim = floatVectorValues.dimension();
+ final float[] quantileGatheringScratch =
+ new float[floatVectorValues.dimension() * Math.min(SCRATCH_SIZE,
totalVectorCount)];
+ int count = 0;
+ double upperSum = 0;
+ double lowerSum = 0;
if (totalVectorCount <= quantizationSampleSize) {
- int copyOffset = 0;
- float[] values = new float[totalVectorCount * dim];
+ int scratchSize = Math.min(SCRATCH_SIZE, totalVectorCount);
+ int i = 0;
while (floatVectorValues.nextDoc() != NO_MORE_DOCS) {
- float[] floatVector = floatVectorValues.vectorValue();
- System.arraycopy(floatVector, 0, values, copyOffset,
floatVector.length);
- copyOffset += dim;
+ float[] vectorValue = floatVectorValues.vectorValue();
+ System.arraycopy(
+ vectorValue, 0, quantileGatheringScratch, i * vectorValue.length,
vectorValue.length);
+ i++;
+ if (i == scratchSize) {
Review Comment:
I think so. If we are sampling these few vectors already, I wouldn't want to
be negatively impacted by extreme confidence intervals & quantiles. Meaning,
extreme confidence intervals can add small biases when calculated over few
points (this is why we do our best to calculate these over `scratchSize * dim`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
