mikemccand commented on issue #15287: URL: https://github.com/apache/lucene/issues/15287#issuecomment-3366839013
> * Faiss recently added [mmap support](https://github.com/facebookresearch/faiss/blame/4fab13c9c67b5402343ca722c83ff7a65a9a48ba/faiss/impl/index_read.cpp#L66-L73) for some indexes, making it possible to read indexes without loading it entirely into RAM -- and we should make use of this functionality wherever possible / desired Cool! Zero copy readers too. I had thought Faiss was RAM heavy, yielding a nice curve from jVector Codec (RAM light), Lucene's default HNSW (nice balance of carefully choosing what should be RAM vs disk), and Faiss (RAM heavy). But it sounds like Faiss is sometimes disk friendly -- these recent mmap improvements, and at least also IVF Flat. > 1. Allow Faiss to use SIMD instructions (more context [here](https://github.com/apache/lucene/blob/602bfbd9af0ee9027de45c1572527eee6b073841/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissLibrary.java#L36-L40)) Wow, that means it's impressive that Faiss shows such good performance in luceneutil's `knnPerfTest.py` when it's using ordinary non-vectorized CPU instructions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
