Pranshu-S opened a new pull request, #16710:
URL: https://github.com/apache/lucene/pull/16710
### Description
The sandbox dedup KNN codec stores each distinct vector once and lets every
document referencing it point to that single shared copy. Its scalar-quantized
variant (DedupScalarQuantizedVectorsFormat) additionally stores a data-blind
scalar-quantized copy of each distinct vector, so scoring runs on the compact
quantized form while the raw vectors remain for full-fidelity readback and
rescoring.
Previously, only FLOAT32 vectors were quantized — FLOAT16 (and BYTE) were
stored raw only. This PR adds scalar-quantization support for FLOAT16 vectors,
bringing them to parity with FLOAT32.
### How
The data-blind scalar quantizer (OptimizedScalarQuantizer) computes
centroids and corrective terms in fp32, and the JVM has no fp16 arithmetic type
— so it currently has no way to quantize fp16 values natively. As a result,
FLOAT16 vectors are inflated short[] → float[] before quantization. This
inflation is lossless (every fp16 value is exactly representable in fp32), so
an fp16 vector produces the same quantized record its fp32 equivalent would.
[This mirrors the core
Lucene104ScalarQuantizedVectorsWriter](https://github.com/apache/lucene/blob/c5fa32ce49f6d28e82ac1b6f111264d955d54a7f/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsWriter.java#L1140).
Quantizing fp16 directly (without inflating) is a follow-up that first
requires the data-blind OptimizedScalarQuantizer itself to support fp16
arithmetic; once that lands, this codec can drop the inflation step. That
prerequisite is tracked by #16533
(https://github.com/apache/lucene/issues/16533).
<!--
If this is your first contribution to Lucene, please make sure you have
reviewed the contribution guide.
https://github.com/apache/lucene/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
