kaivalnp commented on PR #12679: URL: https://github.com/apache/lucene/pull/12679#issuecomment-1812956627
> could you test on cohere with Max-inner product? Thanks, the gist was really helpful and gave some files including normalized and un-normalized vectors. I assume that since you mentioned `MAXIMUM_INNER_PRODUCT`, you wanted the un-normalized vectors I saw \~476k vectors of 768 dimensions there and indexed the first 400k in a *single segment*, while querying the next 10k, using the following command: ```sh ./gradlew :lucene:core:similarity-benchmark --args=" --vecPath=/home/kaivalnp/working/similarity-benchmark/cohere-768.vec --indexPath=/home/kaivalnp/working/similarity-benchmark/cohere-indexes --dim=768 --function=MAXIMUM_INNER_PRODUCT --numDocs=400000 --numQueries=10000 --topKs=5000,2500,1000,500,100 --topK-thresholds=300,305,310,315,320 --traversalSimilarities=295,300,305,310,315 --resultSimilarities=300,305,310,315,320" ``` ### KNN search | maxConn | beamWidth | topK | threshold | count | numVisited | latency | recall | | ------- | --------- | ---- | --------- | ------- | ---------- | ------- | ------ | | 16 | 100 | 5000 | 300.00 | 1123.19 | 40056.44 | 98.96 | 0.89 | | 16 | 100 | 2500 | 305.00 | 480.82 | 23258.29 | 54.91 | 0.83 | | 16 | 100 | 1000 | 310.00 | 191.52 | 11249.93 | 26.12 | 0.73 | | 16 | 100 | 500 | 315.00 | 83.21 | 6487.60 | 14.87 | 0.69 | | 16 | 100 | 100 | 320.00 | 23.80 | 1832.45 | 4.00 | 0.43 | | 16 | 200 | 5000 | 300.00 | 1126.33 | 44928.96 | 107.69 | 0.89 | | 16 | 200 | 2500 | 305.00 | 482.17 | 26242.83 | 61.47 | 0.83 | | 16 | 200 | 1000 | 310.00 | 192.13 | 12751.78 | 29.42 | 0.73 | | 16 | 200 | 500 | 315.00 | 83.49 | 7360.26 | 16.67 | 0.70 | | 16 | 200 | 100 | 320.00 | 23.89 | 2056.14 | 4.51 | 0.44 | | 32 | 100 | 5000 | 300.00 | 1128.81 | 51636.98 | 122.67 | 0.89 | | 32 | 100 | 2500 | 305.00 | 483.29 | 30892.01 | 72.01 | 0.84 | | 32 | 100 | 1000 | 310.00 | 192.65 | 15424.38 | 35.12 | 0.73 | | 32 | 100 | 500 | 315.00 | 83.72 | 9060.78 | 20.28 | 0.70 | | 32 | 100 | 100 | 320.00 | 24.00 | 2606.37 | 5.70 | 0.44 | | 32 | 200 | 5000 | 300.00 | 1130.18 | 61350.93 | 145.76 | 0.89 | | 32 | 200 | 2500 | 305.00 | 483.95 | 37178.70 | 86.05 | 0.84 | | 32 | 200 | 1000 | 310.00 | 192.99 | 18778.34 | 42.14 | 0.73 | | 32 | 200 | 500 | 315.00 | 83.90 | 11083.97 | 24.54 | 0.70 | | 32 | 200 | 100 | 320.00 | 24.08 | 3172.91 | 6.83 | 0.44 | | 64 | 100 | 5000 | 300.00 | 1129.81 | 58389.13 | 138.14 | 0.89 | | 64 | 100 | 2500 | 305.00 | 483.77 | 35567.55 | 81.62 | 0.84 | | 64 | 100 | 1000 | 310.00 | 192.87 | 18093.55 | 40.34 | 0.73 | | 64 | 100 | 500 | 315.00 | 83.84 | 10734.50 | 23.76 | 0.70 | | 64 | 100 | 100 | 320.00 | 24.06 | 3122.13 | 6.77 | 0.44 | | 64 | 200 | 5000 | 300.00 | 1130.78 | 72620.92 | 169.86 | 0.89 | | 64 | 200 | 2500 | 305.00 | 484.24 | 45052.36 | 101.93 | 0.84 | | 64 | 200 | 1000 | 310.00 | 193.16 | 23283.96 | 51.61 | 0.73 | | 64 | 200 | 500 | 315.00 | 83.99 | 13908.95 | 30.44 | 0.70 | | 64 | 200 | 100 | 320.00 | 24.13 | 4035.89 | 8.61 | 0.44 | ### Similarity-based search | maxConn | beamWidth | traversalSimilarity | resultSimilarity | count | numVisited | latency | recall | | ------- | --------- | ------------------- | ---------------- | ------- | ---------- | ------- | ------ | | 16 | 100 | 295.00 | 300.00 | 1209.53 | 18270.70 | 44.38 | 0.95 | | 16 | 100 | 300.00 | 305.00 | 538.00 | 8833.17 | 21.02 | 0.93 | | 16 | 100 | 305.00 | 310.00 | 239.11 | 4249.13 | 9.97 | 0.91 | | 16 | 100 | 310.00 | 315.00 | 105.02 | 2050.95 | 4.87 | 0.87 | | 16 | 100 | 315.00 | 320.00 | 45.71 | 1028.26 | 2.35 | 0.83 | | 16 | 200 | 295.00 | 300.00 | 1217.74 | 20335.62 | 49.38 | 0.96 | | 16 | 200 | 300.00 | 305.00 | 542.19 | 9851.65 | 23.54 | 0.94 | | 16 | 200 | 305.00 | 310.00 | 240.68 | 4726.50 | 11.04 | 0.91 | | 16 | 200 | 310.00 | 315.00 | 106.02 | 2287.34 | 5.33 | 0.88 | | 16 | 200 | 315.00 | 320.00 | 46.09 | 1139.68 | 2.60 | 0.84 | | 32 | 100 | 295.00 | 300.00 | 1235.75 | 25159.18 | 59.94 | 0.98 | | 32 | 100 | 300.00 | 305.00 | 554.76 | 12709.10 | 29.69 | 0.96 | | 32 | 100 | 305.00 | 310.00 | 247.15 | 6275.45 | 14.46 | 0.94 | | 32 | 100 | 310.00 | 315.00 | 108.95 | 3093.07 | 7.00 | 0.91 | | 32 | 100 | 315.00 | 320.00 | 47.39 | 1544.48 | 3.47 | 0.86 | | 32 | 200 | 295.00 | 300.00 | 1243.78 | 29690.87 | 70.66 | 0.98 | | 32 | 200 | 300.00 | 305.00 | 558.98 | 15064.99 | 34.99 | 0.97 | | 32 | 200 | 305.00 | 310.00 | 249.03 | 7442.06 | 17.09 | 0.95 | | 32 | 200 | 310.00 | 315.00 | 110.01 | 3664.88 | 8.28 | 0.92 | | 32 | 200 | 315.00 | 320.00 | 47.92 | 1826.35 | 4.06 | 0.87 | | 64 | 100 | 295.00 | 300.00 | 1228.98 | 29028.54 | 68.77 | 0.97 | | 64 | 100 | 300.00 | 305.00 | 549.09 | 14931.68 | 34.43 | 0.95 | | 64 | 100 | 305.00 | 310.00 | 242.41 | 7417.15 | 16.89 | 0.92 | | 64 | 100 | 310.00 | 315.00 | 105.26 | 3613.84 | 8.12 | 0.88 | | 64 | 100 | 315.00 | 320.00 | 45.14 | 1794.89 | 4.02 | 0.82 | | 64 | 200 | 295.00 | 300.00 | 1243.45 | 36266.02 | 85.05 | 0.98 | | 64 | 200 | 300.00 | 305.00 | 557.47 | 18811.49 | 42.83 | 0.96 | | 64 | 200 | 305.00 | 310.00 | 246.42 | 9377.28 | 21.11 | 0.94 | | 64 | 200 | 310.00 | 315.00 | 107.09 | 4559.22 | 10.20 | 0.89 | | 64 | 200 | 315.00 | 320.00 | 45.99 | 2249.22 | 4.99 | 0.84 | **IF** the goal is to "get all vectors above a similarity", then looks like using the new similarity-based search API scales better than having a large `topK` and post-filtering results later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
