benwtrent commented on PR #14178: URL: https://github.com/apache/lucene/pull/14178#issuecomment-2703746035
> @benwtrent @navneet1v I wonder if either of you were able to replicate benchmarks? I didn't want to leave you hanging @kaivalnp, especially after you have obviously put a ton of good work in this codec. I haven't had a chance to benchmark and I likely won't. I personally don't want to bother messing with FAISS. No offense meant. I think the work put in here is pretty cool. Putting it in the sandbox, as long as building sandbox doesn't require a new dependency (e.g. folks who don't want to bother with FAISS don't have to worry about downloading it when building), is probably ok. I like the utilization of memory segments and modern JVM features. Shows the power of the JVM and the flexibility of Lucene's codec plugin system. Just like the CAGRA index that is being worked on in another PR (yet another cool experiment that will likely never be put in Lucene core). While I think the performance numbers are cool, they indicate that this doesn't actually buy us that much and the cost of bringing in a large dependency like FAISS is huge. This is especially the case where most of the critical logic is done in an memory unsafe language (looking at you c++). There is still a ton of ground to gain in Lucene core (e.g. graph storage & searching optimization, multi-graph search, merging, etc.). And while some of that work isn't "sexy", it will likely help the Lucene community as a whole much more. In short, I am philosophically against having a dependency against FAISS in Lucene core, though having a neat example codec that utilizes modern JVM features in sandbox is pretty cool. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
