kaivalnp commented on code in PR #16710:
URL: https://github.com/apache/lucene/pull/16710#discussion_r4148777164
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupScalarQuantizedVectorValues.java:
##########
@@ -328,4 +383,88 @@ public VectorScorer rescorer(float[] target) throws
IOException {
return rawValues.rescorer(target);
}
}
+
+ /**
+ * FLOAT16 analogue of {@link RawAndQuantizedValues}: exposes a field's raw
de-duplicated {@code
+ * short[]} vectors for full-fidelity readback, while {@link
#scorer(short[])} scores against the
+ * shared quantized view (the {@code short[]} target is inflated to {@code
float[]} to match the
+ * data-blind quantizer). Mirrors the FLOAT16 handling in {@code
+ * Lucene104ScalarQuantizedVectorsReader}.
+ */
+ static final class Float16RawAndQuantizedValues extends Float16VectorValues
+ implements DedupVectorValues {
+ private final DedupVectorValues.Float16Impl rawValues;
+ private final FieldValues quantizedValues;
+
+ Float16RawAndQuantizedValues(
+ DedupVectorValues.Float16Impl rawValues, FieldValues quantizedValues) {
+ this.rawValues = rawValues;
+ this.quantizedValues = quantizedValues;
+ }
+
+ FieldValues getQuantizedValues() {
+ return quantizedValues;
+ }
+
+ @Override
+ public KnnVectorValues getGroupView() {
+ return rawValues.getGroupView();
+ }
+
+ @Override
+ public FieldOrdToGroupOrd getFieldOrdToGroupOrd() {
+ return rawValues.getFieldOrdToGroupOrd();
+ }
+
+ @Override
+ public int dimension() {
+ return rawValues.dimension();
+ }
+
+ @Override
+ public int size() {
+ return rawValues.size();
+ }
+
+ @Override
+ public int ordToDoc(int ord) {
+ return rawValues.ordToDoc(ord);
+ }
+
+ @Override
+ public Bits getAcceptOrds(Bits acceptDocs) {
+ return rawValues.getAcceptOrds(acceptDocs);
+ }
+
+ @Override
+ public void prefetch(int[] ordsToPrefetch, int numOrds) throws IOException
{
+ rawValues.prefetch(ordsToPrefetch, numOrds);
+ }
+
+ @Override
+ public short[] vectorValue(int ord) throws IOException {
+ return rawValues.vectorValue(ord);
+ }
+
+ @Override
+ public DocIndexIterator iterator() {
+ return rawValues.iterator();
+ }
+
+ @Override
+ public Float16RawAndQuantizedValues copy() throws IOException {
+ return new Float16RawAndQuantizedValues(rawValues.copy(),
quantizedValues.copy());
+ }
+
+ @Override
+ public VectorScorer scorer(short[] target) throws IOException {
+ float[] inflated = DedupUtil.inflateFloat16(target, new
float[target.length]);
Review Comment:
The
[`BaseQuantizedByteVectorValues`](https://github.com/apache/lucene/blob/5e537615b292d5af7cd38c7546842f76dc1fba66/lucene/core/src/java/org/apache/lucene/util/quantization/BaseQuantizedByteVectorValues.java#L43-L51)
does have a scorer on a `short[] query`, which is what implementers like
[`OffHeapScalarQuantizedFloat16VectorValues`](https://github.com/apache/lucene/blob/5e537615b292d5af7cd38c7546842f76dc1fba66/lucene/core/src/java/org/apache/lucene/codecs/lucene104/OffHeapScalarQuantizedFloat16VectorValues.java#L238-L239)
use.
Perhaps we need to make `FieldValues` `abstract sealed`, with specific impls
for fp32 and fp16?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
