benwtrent commented on issue #12615:
URL: https://github.com/apache/lucene/issues/12615#issuecomment-1802705393
@kevindrosendahl if I am reading the code correctly, it does the following:
- Write int8 quantized vectors along side the vector ordinals in the graph
(`.vex` or whatever has a copy of each vector).
- Continue to write vectors in `.vec`. I am guessing this is a stop-gap and
you are thinking of removing this? Maybe not?
I have a concern around index sorting. How does building the graph &
subsequent `getFloatVectors()` play with index sorting? Usually when folks sort
an index, they expect to be able to iterate values in that given order. Is it
horrifically slow to iterate `.vex` in this scenario?
What do we think about always keeping `.vec` around? Probably should for
re-ranking purposes once more extreme quantization measures are used.
One more question, have you tested your implementation in the situation
where `.vex` cannot all be paged into memory and it faired ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
