benwtrent commented on issue #13350: URL: https://github.com/apache/lucene/issues/13350#issuecomment-2122846701
I used int7 for my experiments. While losing one bit of precision isn't the best, it works well. I explored adding an unsigned byte dot product, but that got rejected as too much code. I think for int8 to support all vector similarities, we need an unsigned dot product. Additionally, if we support int8, we should restrict it to [-127, 127], as there can be nice performance benefits if we can assume these ranges on various hardware. I haven't done the math on Euclidean to figure out if we need an unsigned byte version of that as well. I am out on vacation. But here is my old PR: https://github.com/apache/lucene/pull/12694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
