mikemccand commented on issue #13519: URL: https://github.com/apache/lucene/issues/13519#issuecomment-2315154624
I just tested KNN recall using [`knnPerfTest.py` from `luceneutil`](https://github.com/mikemccand/luceneutil/blob/main/src/python/knnPerfTest.py) on 4, 7, 8 bit quantization, and still see 8 bit quantization broken. This is with Cohere (768 dimension) vectors, 250K docs, 32 `maxConn`, 50 `beamWidthIndex`, 20 `fanout`. For EUCLIDEAN: ``` recall latency nDoc fanout maxConn beamWidth quantized visited index ms selectivity filterType 0.541 1.27 250000 20 32 50 4 bits 7156 18786 1.00 post-filter 0.886 1.18 250000 20 32 50 7 bits 6763 17791 1.00 post-filter 0.038 1.74 250000 20 32 50 8 bits 10066 26265 1.00 post-filter ``` And `DOT_PRODUCT` (angular): ``` recall latency nDoc fanout maxConn beamWidth quantized visited index ms selectivity filterType 0.497 0.96 250000 20 32 50 4 bits 4903 16632 1.00 post-filter 0.771 0.87 250000 20 32 50 7 bits 4319 15565 1.00 post-filter 0.003 0.92 250000 20 32 50 8 bits 9157 30284 1.00 post-filter ``` And `COSINE`: ``` recall latency nDoc fanout maxConn beamWidth quantized visited index ms selectivity filterType 0.531 1.23 250000 20 32 50 4 bits 6816 20618 1.00 post-filter 0.650 1.22 250000 20 32 50 7 bits 6921 19454 1.00 post-filter 0.002 1.00 250000 20 32 50 8 bits 8692 188290 1.00 post-filter ``` Should we maybe just remove 8 bit support? From the discussion above it sounds like even the fixes we are testing are not much better than 7 bit, and add substantial code complexity? In any event, I think this should be a blocker for 9.12 / 10.0? We should do something before releasing (fix 8 bit case, or remove it)... (It's also entirely possible I am making some sort of silly mistake trying to run this tooling that I do not fully understand, heh). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
