benwtrent commented on PR #13288: URL: https://github.com/apache/lucene/pull/13288#issuecomment-2149954395
@navneet1v > Does this mean now Lucene supports BitVectorsFormat officially? The answer is no. > Or it was more a prototype and not intended for production use? The answer is yes. > what is the point of VectorSimilarity functions in case of bitvectors format. Currently there is none. But I could see it being updated where cosine and dot-product aren't actually just hamming distance (as hamming is more akin to euclidean). > So it means getting setting vector similarity from a field is not the source truth for what vector similarity function to be used. For the default and core codecs, keeping a nice separation so that users don't have to know about the codec and trusting it is doing the right thing is important. Using the similarity in Field Info allows users to have a pick of some default supported vector similarity functions without futzing around with codecs (which is complicated for normal Lucene users). It is important for ease of use. As for the format summarily ignoring the input, this could always be done. The format stores, reads, scores, etc. any way it wants. If the advance user chooses a custom format that ignores the similarity applied to the field, its their prerogative. For example, its conceptual that a format could actually ignore cosine altogether and always normalize, store the magnitude, and always do dot-product. I do not think the bit-vector format necessitates a different contract between vector similarities and formats. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
