benwtrent commented on issue #13403: URL: https://github.com/apache/lucene/issues/13403#issuecomment-2143529235
I agree @mikemccand, we should not do any dim-reduction stuff until some threshold of vectors is reached. I am not 100% convinced this scales nicely, well, it would scale nicer than running PQ on tiny segments all the time :). We already have some expensive issues with HNSW merging (we are slowly working through fixing those). When segments are merged that have different PQ code-books, we have to re-calculate and re-quantize everything unless we can do something clever. Maybe it won't be too expensive and the cost will be worth it if we can build the HNSW graph with the PQ or maybe use the PQ information to boot strap the HNSW graph build to make it cheaper. I agree this is a worthy place for experimentation. As an aside, this "wait to build the index" thing could also be done for HNSW. Tiny segments with quick flushes probably shouldn't even build HNSW graphs. Instead, they should probably store the float vectors flat (or the scalar quantized vectors flat as scalar quantizing is effectively linear in runtime). Then when a threshold is reached (it could be small, something like 1k, 10k?), we create the HNSW graphs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
