benwtrent commented on code in PR #14304:
URL: https://github.com/apache/lucene/pull/14304#discussion_r2012222706
##########
lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:
##########
@@ -334,4 +334,45 @@ public static int findNextGEQ(int[] buffer, int target,
int from, int to) {
assert IntStream.range(0, to - 1).noneMatch(i -> buffer[i] > buffer[i +
1]);
return IMPL.findNextGEQ(buffer, target, from, to);
}
+
+ /**
+ * Quantizes {@code vector}, putting the result into {@code dest}.
+ *
+ * @param vector the vector to quantize
+ * @param dest the destination vector, can be null
+ * @param scale the scaling factor
+ * @param alpha the alpha value
+ * @param minQuantile the lower quantile of the distribution
+ * @param maxQuantile the upper quantile of the distribution
+ * @return the corrective offset that needs to be applied to the score
+ */
+ public static float quantize(
Review Comment:
lets unify the name here with the implementation
##########
lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:
##########
@@ -334,4 +334,45 @@ public static int findNextGEQ(int[] buffer, int target,
int from, int to) {
assert IntStream.range(0, to - 1).noneMatch(i -> buffer[i] > buffer[i +
1]);
return IMPL.findNextGEQ(buffer, target, from, to);
}
+
+ /**
+ * Quantizes {@code vector}, putting the result into {@code dest}.
+ *
+ * @param vector the vector to quantize
+ * @param dest the destination vector, can be null
+ * @param scale the scaling factor
+ * @param alpha the alpha value
+ * @param minQuantile the lower quantile of the distribution
+ * @param maxQuantile the upper quantile of the distribution
+ * @return the corrective offset that needs to be applied to the score
+ */
+ public static float quantize(
+ float[] vector, byte[] dest, float scale, float alpha, float
minQuantile, float maxQuantile) {
+ assert vector.length == dest.length;
Review Comment:
Let's throw an actual error here, illegal argument?
##########
lucene/core/src/java/org/apache/lucene/internal/vectorization/DefaultVectorUtilSupport.java:
##########
@@ -234,4 +234,79 @@ public static long int4BitDotProductImpl(byte[] q, byte[]
d) {
}
return ret;
}
+
+ @Override
+ public float minMaxScalarQuantize(
+ float[] vector, byte[] dest, float scale, float alpha, float
minQuantile, float maxQuantile) {
+ return new ScalarQuantizer(alpha, scale, minQuantile,
maxQuantile).quantize(vector, dest, 0);
+ }
+
+ @Override
+ public float recalculateScalarQuantizationOffset(
+ byte[] vector,
+ float oldAlpha,
+ float oldMinQuantile,
+ float scale,
+ float alpha,
+ float minQuantile,
+ float maxQuantile) {
+ return new ScalarQuantizer(alpha, scale, minQuantile, maxQuantile)
+ .recalculateOffset(vector, 0, oldAlpha, oldMinQuantile);
+ }
+
+ static class ScalarQuantizer {
Review Comment:
private?
##########
lucene/core/src/java/org/apache/lucene/util/VectorUtil.java:
##########
@@ -334,4 +334,45 @@ public static int findNextGEQ(int[] buffer, int target,
int from, int to) {
assert IntStream.range(0, to - 1).noneMatch(i -> buffer[i] > buffer[i +
1]);
return IMPL.findNextGEQ(buffer, target, from, to);
}
+
+ /**
+ * Quantizes {@code vector}, putting the result into {@code dest}.
Review Comment:
Scalar quantizes.
##########
lucene/core/src/java21/org/apache/lucene/internal/vectorization/PanamaVectorUtilSupport.java:
##########
@@ -907,4 +907,97 @@ public static long int4BitDotProduct128(byte[] q, byte[]
d) {
}
return subRet0 + (subRet1 << 1) + (subRet2 << 2) + (subRet3 << 3);
}
+
+ @Override
+ public float minMaxScalarQuantize(
+ float[] vector, byte[] dest, float scale, float alpha, float
minQuantile, float maxQuantile) {
+ float correction = 0;
Review Comment:
lets add an assert here on vector.length & dest.length. Earlier up stream,
we should throw an actual production error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
