Pranshu-S commented on code in PR #16710:
URL: https://github.com/apache/lucene/pull/16710#discussion_r4143902705
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupScalarQuantizedVectorValues.java:
##########
@@ -328,4 +383,88 @@ public VectorScorer rescorer(float[] target) throws
IOException {
return rawValues.rescorer(target);
}
}
+
+ /**
+ * FLOAT16 analogue of {@link RawAndQuantizedValues}: exposes a field's raw
de-duplicated {@code
+ * short[]} vectors for full-fidelity readback, while {@link
#scorer(short[])} scores against the
+ * shared quantized view (the {@code short[]} target is inflated to {@code
float[]} to match the
+ * data-blind quantizer). Mirrors the FLOAT16 handling in {@code
+ * Lucene104ScalarQuantizedVectorsReader}.
+ */
+ static final class Float16RawAndQuantizedValues extends Float16VectorValues
+ implements DedupVectorValues {
+ private final DedupVectorValues.Float16Impl rawValues;
+ private final FieldValues quantizedValues;
+
+ Float16RawAndQuantizedValues(
+ DedupVectorValues.Float16Impl rawValues, FieldValues quantizedValues) {
+ this.rawValues = rawValues;
+ this.quantizedValues = quantizedValues;
+ }
+
+ FieldValues getQuantizedValues() {
+ return quantizedValues;
+ }
+
+ @Override
+ public KnnVectorValues getGroupView() {
+ return rawValues.getGroupView();
+ }
+
+ @Override
+ public FieldOrdToGroupOrd getFieldOrdToGroupOrd() {
+ return rawValues.getFieldOrdToGroupOrd();
+ }
+
+ @Override
+ public int dimension() {
+ return rawValues.dimension();
+ }
+
+ @Override
+ public int size() {
+ return rawValues.size();
+ }
+
+ @Override
+ public int ordToDoc(int ord) {
+ return rawValues.ordToDoc(ord);
+ }
+
+ @Override
+ public Bits getAcceptOrds(Bits acceptDocs) {
+ return rawValues.getAcceptOrds(acceptDocs);
+ }
+
+ @Override
+ public void prefetch(int[] ordsToPrefetch, int numOrds) throws IOException
{
+ rawValues.prefetch(ordsToPrefetch, numOrds);
+ }
+
+ @Override
+ public short[] vectorValue(int ord) throws IOException {
+ return rawValues.vectorValue(ord);
+ }
+
+ @Override
+ public DocIndexIterator iterator() {
+ return rawValues.iterator();
+ }
+
+ @Override
+ public Float16RawAndQuantizedValues copy() throws IOException {
+ return new Float16RawAndQuantizedValues(rawValues.copy(),
quantizedValues.copy());
+ }
+
+ @Override
+ public VectorScorer scorer(short[] target) throws IOException {
+ float[] inflated = DedupUtil.inflateFloat16(target, new
float[target.length]);
Review Comment:
Yes. Even when the underlying view is fp16, the [scorer only accepts
float[]](https://github.com/apache/lucene/blob/main/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupScalarQuantizedVectorValues.java#L241-L249)
— that's the current data-blind limitation. If the scorer used in FieldValues
exposed a scorer(short[]) that natively quantized fp16, we could pass the
short[] straight through; it doesn't, so the query must be inflated first.
I initially thought of moving this inflation in FieldValues itself but on
second that, that is the encoding-agnostic quantized view, shared by the
FLOAT32 wrapper too. Adding scorer(short[]) there would imply "the quantized
view natively supports fp16 queries," when in reality it would just inflate to
fp32 internally which feels semantically wrong.
Let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
