Pranshu-S commented on issue #16029:
URL: https://github.com/apache/lucene/issues/16029#issuecomment-6918788909
Hey @mccullocht @kaivalnp, I was trying to see what would be the cleanest
way to unblock data-blind quantization, as it was reverted earlier due to how
it behaves when merges read segments through reader wrappers, which results in
behaviour discrepancies. This also sort of raises a wider question on how the
updated codec will behave with wrappers and merges, as well as backward
compatibility.
I had my two cents on it and wanted a wider opinion before I commit to doing
the change.
**The problem**
In the original PR, `Lucene104ScalarQuantizedVectorsWriter` decided how to
merge each input segment by checking whether it still has its original floats.
However, that check only works when the input unwraps to a
`Lucene104ScalarQuantizedVectorsReader`. Otherwise it treats the input as
another format and checks `getFloatVectorValues().size() > 0`. A data-blind
segment answers that with floats reconstructed from its quantized bytes, so the
writer concludes the original floats exist.
Merges wrap inputs in several cases:
- `SlowCodecReaderWrapper`, from merge policies that wrap inputs
- `SortingCodecReader` over `SlowCompositeCodecReaderWrapper`, for
reordering merges
- `DocRangeKnnVectorsReader`, for partitioned merges
When that happens: a segment without floats can be merged into a `CENTERED`
or `DATA_BLIND_WITH_FLOATS` output, which then stores reconstructions as if
they were the original vectors
If we look closely, the problem is not the code but the fact that Lucene
works with the assumption that every format's merge fallback is lossless: when
a merge doesn't recognize an input, it just reads the values back through the
standard API and writes them again. So failing to recognize an input only makes
the merge slower in the fallback cases.
`DATA_BLIND_WITHOUT_FLOATS` is the first mode where this is NOT true by
design, since we don't keep the original vectors at all.
One way would be to update the wrappers to expose what they wrap. But
wrappers are also something users can write themselves, and existing ones
wouldn't forward a new method, so the problem would quietly remain for them.
**Proposal**
If we honestly think about it, since the quantized vector is standalone,
without any backing float vector, it can be considered a derived encoding of
its own. Doing it this way also makes sense because every codec and wrapper
then has to handle it explicitly, instead of being handed floats that look like
the originals. Since data-blind quantizes against a zero centroid, the
quantized record doesn't depend on the segment, so it can be the field's stored
value.
A new `VectorEncoding` (e.g. `QUANTIZED`) would store the records as the
field's real values, and floats would only be input at index time. `CENTERED`
and `DATA_BLIND_WITH_FLOATS` would stay codec modes of float fields, since they
keep the original floats and don't have this problem.
Pros:
- Merges are lossless by design. Wrappers and generic merges move the
records the same way they already move `BYTE`, `FLOAT32` and `FLOAT16`, so
reordering and partitioned merges just work.
- The one-way-door rules (no re-encoding, no re-centering, the same encoding
across segments) become schema rules, checked at indexing time instead of
failing during merge.
- Reconstructions are never passed off as originals, so converting to other
formats can't silently degrade the vectors.
- The compiler enforces it: every `switch` over `VectorEncoding` has to
handle the new value, so no code path can quietly treat these vectors as plain
floats.
Cons:
- A big core change, similar in size to adding `FLOAT16`
- Quantization becomes a schema choice instead of a codec setting, so moving
an existing field means a new field or a reindex (which is also the right
expectation in my opinion).
Would love to hear what others think, especially whether a new encoding is
acceptable here, or if we'd rather look at something lighter for now (e.g. a
per-segment flag in `FieldInfo` attributes that merges check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]