Pranshu-S opened a new pull request, #16706:
URL: https://github.com/apache/lucene/pull/16706
### Description
The dedup KNN codec stores each distinct vector once and lets every document
that uses it point to that single shared copy. This results in two forms of
ordinals -
- A field ordinal identifies the n-th document (that has a vector) in a
field — one per document.
- A group ordinal identifies the n-th distinct vector within a group — one
per unique vector.
Previously, this mapping between field and group was bit-packed at a
hardcoded 32 bits per value. However, because de-duplication keeps the number
of distinct vectors (group ordinals) small relative to the number of documents,
group ordinals would likely be below 2³², so most of those 32 bits were wasted
on every document of every field.
This PR packs each field's map at the minimum DirectWriter bit width
needed to represent that field's largest group ordinal, and persists the chosen
width to the metadata so the reader can decode it.
Addresses one of the optimisations identified in
https://github.com/apache/lucene/issues/16595
<!--
If this is your first contribution to Lucene, please make sure you have
reviewed the contribution guide.
https://github.com/apache/lucene/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
