shubhamvishu commented on PR #14963: URL: https://github.com/apache/lucene/pull/14963#issuecomment-3423555880
@benwtrent I reran the benchmarks and somehow I don't see the old performance difference like earlier and rather similar to baseline. Maybe the recent developments had some indexing improvements (like eg I see 4 bit recall is much better now due to OSQ?). Net-net, against the current main we are not creating those many tiny segments in our luceneutil benchmarks by default. Maybe we see still see higher benefits in older Lucene version probably? However, to simulate a more realtime update scenario by changing luceneutil `KnnIndexer` to use [setMaxBufferedDocs(100)](https://github.com/mikemccand/luceneutil/blob/main/src/main/knn/KnnIndexer.java#L102) to trigger more flushes and re-ran the benchmarks and I see the below results where indexing gets **~30%** faster with 1000 as the threshold(configurable) in that particular case slightly . So it seems the user could configure a value that helps avoid the overhead of creating graphs. @benwtrent @ChrisHegarty I patched the test change in the latest commit, let me know if this looks good or not. Thanks! **Baseline :** ``` Results: recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn beamWidth quantized visited index(s) index_docs/s num_segments index_size(MB) vec_disk(MB) vec_RAM(MB) indexType 0.901 5.941 5.923 0.997 500000 100 50 64 250 4 bits 31624 84.64 5907.09 4 1690.88 1655.579 190.735 HNSW 0.965 13.023 13.012 0.999 500000 100 50 64 250 7 bits 40750 127.19 3931.00 7 1871.98 1838.684 373.840 HNSW 0.971 9.997 9.986 0.999 500000 100 50 64 250 no 40086 103.35 4837.93 7 1498.13 1464.844 1464.844 HNSW ``` **Candidate (HNSW_GRAPH_THRESHOLD = 100) [DEFAULT]:** ``` Results: recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn beamWidth quantized visited index(s) index_docs/s num_segments index_size(MB) vec_disk(MB) vec_RAM(MB) indexType 0.898 4.598 4.587 0.998 500000 100 50 64 250 4 bits 23503 88.53 5647.55 2 1692.61 1655.579 190.735 HNSW 0.964 13.309 13.291 0.999 500000 100 50 64 250 7 bits 41490 122.04 4097.02 8 1871.81 1838.684 373.840 HNSW 0.964 7.809 7.788 0.997 500000 100 50 64 250 no 27908 104.95 4764.08 3 1501.41 1464.844 1464.844 HNSW ``` **Candidate (HNSW_GRAPH_THRESHOLD = 1000) :** ``` Results: recall latency(ms) netCPU avgCpuCount nDoc topK fanout maxConn beamWidth quantized visited index(s) index_docs/s num_segments index_size(MB) vec_disk(MB) vec_RAM(MB) indexType 0.901 6.737 6.720 0.997 500000 100 50 64 250 4 bits 37284 58.96 8480.61 6 1689.44 1655.579 190.735 HNSW 0.959 10.027 10.015 0.999 500000 100 50 64 250 7 bits 31131 86.87 5755.46 4 1874.15 1838.684 373.840 HNSW 0.963 7.463 7.452 0.999 500000 100 50 64 250 no 27939 72.11 6933.85 3 1501.16 1464.844 1464.844 HNSW ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
