RKSPD commented on PR #16709:
URL: https://github.com/apache/lucene/pull/16709#issuecomment-7026920057

   > @RKSPD It'd be great to push this forwards! This would be a breath of 
fresh air compared to HNSW heavy options Lucene has today (actually, Lucene 
also has 
[`FaissKnnVectorsFormat`](https://github.com/apache/lucene/blob/branch_11x/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissKnnVectorsFormat.java)
 , which itself has many options via its [cool index factory 
string](https://github.com/facebookresearch/faiss/wiki/The-index-factory)), a 
completely different vector engine, quickly pluggable via the `Codec` 
(`KnnVectorsFormat`) API. It'd give options to Lucene users, and also enhance 
our testing since we could pick vector search from these two diverse options.
   > 
   > Also, since this is targeting Lucene sandbox module, where things are 
allowed to be sandy, we shouldn't expect/aim to perfect/polish this PR so much 
-- it just needs to be basically functional as a starting point, and 
enhancements come later: progress not perfection.
   
   Hi Mike, thanks for your support! I've updated the PR with some quality of 
life improvements that hopefully makes SegmentIVF behave better at scale and 
easier to benchmark. There are a lot of cool features here that I hope I can 
bring to the broader community. The codec behaves a little differently to HNSW, 
namely it does a lot more work per query at similar recall, but brings down the 
latency with sequential access and tightly coupled quantizer design. It also 
has a simpler way to tradeoff RAM or latency for recall, with either 
`spillBits` or `nprobe` tuneables. It would be really interesting to see 
SegmentIVF in more benchmarks for LKNN performance!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to