ChrisHegarty commented on PR #13076: URL: https://github.com/apache/lucene/pull/13076#issuecomment-1932304149
>One difficulty is making sure the SPI interface is one we want (seems float[] & byte[] is too restrictive?). Some other work from @ChrisHegarty (https://github.com/apache/lucene/pull/12703) shows that we may want to move away from float[], float[] and potentially an interface like. > >score(int vectorOrdinal1, int vectorOrdinal2) score(float[] queryVector, int vectorOrdinal) score(byte[] queryVector, int vectorOrdinal). I've been experimenting with various potential optimisations and variants for some of these distance computations, and also pushing on some limitations of the Panama Vector API (in part to feedback into the JDK API). We should really be able to compare these things either on of off heap. In that way, I agree with the comparison function being `score(int ord1, int ord2)`. It should not matter where the data actually is. Lucene core should **not** try to handle all possible variants of bit-size and distance function combinations, but rather support a subset of such along with the ability to extend. Extension allows different folk to experiment more easily in this evolving area - this is effectively what I'm doing locally. Successful and interesting experiments, when proven, can then be proposed separately on their own merit, maybe as a Lucene extension or misc package or maybe not at all. Ultimately, Lucene should benefit from "best in class" here, but not have to accept each and every variant into core. The addition of well thought out minimal extension points - for scoring - would be of long term benefit to Lucene. I'm happy to work on such, since I've been hacking around this area locally for a while now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
