jimczi commented on issue #14681: URL: https://github.com/apache/lucene/issues/14681#issuecomment-2987729521
Thanks for opening this and for all the work going into expanding vector search in Lucene. That said, I’m a bit concerned about the growing number of options being proposed that seem to have similar trade-offs, just implemented differently. We’ve added Faiss recently, and now there’s talk of bringing in another graph-based method that doesn’t really change the fundamental limitations we already have with the HNSW codec. Things like Vamana or product quantization can definitely be useful, but from what I can tell, they offer similar properties to what we already have with HNSW + BBQ. It feels like we’re piling on more alternatives without really addressing the pain points some users face with graph-based indexing in general. Also, calling this a “disk-based” solution seems a bit misleading if the graph still has to be built fully in memory. That’s often the core problem people are trying to get around. Instead, I think it might be more valuable to dig into the specific improvements JVector brings and see if any of those could make our current HNSW implementation better. I thought that was already explored in [#12615](https://github.com/apache/lucene/issues/12615), and as I recall, there wasn’t a strong enough differentiator to justify pulling in the full JVector approach. Revisiting that might be a better path forward. I totally get that OpenSearch wants to contribute their work upstream, and that’s great to see. It just feels important that we avoid overwhelming users with too many similar graph-based options, especially when they don’t clearly solve new problems. Appreciate the ongoing discussion and all the effort here! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
