sam-herman commented on issue #14681: URL: https://github.com/apache/lucene/issues/14681#issuecomment-2993767101
> Things like Vamana or product quantization can definitely be useful, but from what I can tell, they offer similar properties to what we already have with HNSW + BBQ. jVector is actually a combination of both Vamana style and HNSW hierarchy combined in the same graph with the option of using PQ or BQ construction/search. In terms of properties there are some unique functionalities, to name a few: 1. **NVQ (Non-Linear vector Quantization)** - on-disk vector format which we will add to the plugin pretty soon as well. It exists in jVector but we haven't made the integration to the plugin just yet, but coming pretty soon. 2. **Inline vectors** - Both separate (NVQ or FP) and inline storage formats for vectors that improve IO patterns. 3. **Concurrency** - jVector allows for concurrent build of graph index _Note: there are upcoming additional differentiators that will be released soon as well, and can update once those are released._ > Also, calling this a “disk-based” solution seems a bit misleading if the graph still has to be built fully in memory. That’s often the core problem people are trying to get around. if the FP vectors are not stored in memory we have noticed that the graph structure is overall pretty lean and can fit on the JVM heap pretty easily, even on low heaps. Moreover, another important factor is that of efficient IO access for scoring FP vectors even for the non quantized use cases. A few months back I noticed significantly more IO access in the Lucene HNSW format than the jVector version at the time. > Instead, I think it might be more valuable to dig into the specific improvements JVector brings and see if any of those could make our current HNSW implementation better. I thought that was already explored in https://github.com/apache/lucene/issues/12615, and as I recall, there wasn’t a strong enough differentiator to justify pulling in the full JVector approach. Revisiting that might be a better path forward. I think this is not much different than FAISS integration in the sense that it is possible to try and copy all the code over to Lucene HNSW implementation. That however is not always possible and even when it is, it can create maintenance difficulties. Consider the scenario of DataStax that uses jVector in a number of projects such as C* and OpenSearch. Ideally we would not want to maintain copies of the same code in various projects and rather have an easy path to leverage the innovation across projects. Integration of the jVector codec to Lucene helps with the problem of portability, as it doesn't require the overhead of maintaining OpenSearch plugin and allows for a more seamless integration. At the same time it allows to shift more resources to innovation without blocking or hindering other approaches in other codecs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
