kaivalnp commented on PR #15784: URL: https://github.com/apache/lucene/pull/15784#issuecomment-5405018536
Hi @hossman, yes -- this change was intentional, and it is indeed unfortunate that the API is the same :( To add more context: this query attempts to find all (approximate) vectors scoring above `resultSimilarity` (added in #12679), which can be visualized like a radius query in Euclidean space (equivalent to "find all vectors within a radius"). Prior to Lucene 10.5, the query found these approximate vectors by traversing all nodes in the HNSW graph above an explicit `traversalSimilarity` (i.e. bigger radius), but this parameter was dataset / query sensitive and had its issues (e.g. it had to be higher for queries in a sparse part of the vector space, lower in dense parts -- leading to unnecessary cost in some queries, and missed results in others). As a consequence, we explored this adaptive traversal strategy, where the similarity for graph traversal (i.e. equivalent of `traversalSimilarity`) starts with a low value and "decay"s towards `resultSimilarity` on encountering lower-scoring nodes (with a factor of `decay`). This algorithm gave a [non-trivially better recall / latency tradeoff](https://github.com/apache/lucene/pull/15784#issuecomment-4042688192) on multiple open-source and internal datasets I tested it on. As for the migration, the `resultSimilarity` parameter is exactly the same as before (i.e. find all vectors scoring above this threshold). For `traversalSimilarity` there is no 1:1 equivalent, and the `decay` parameter is directionally inverse (i.e. higher `decay` leads to more exploration in the HNSW graph and better quality results, while lower `decay` leads to lesser exploration and more approximate results). I'd suggest starting with the default of `decay = 0.5f`, and increasing / decreasing the value based on the target quality of results (i.e. recall against exact radius hits) that you were observing prior to the Lucene upgrade. You can use the popular open-source Lucene KNN benchmark (https://github.com/mikemccand/luceneutil) for this tuning (with `searchType = radius`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
