Re: [PR] Speedup merging of HNSW graphs [lucene]

[via GitHub]

mayya-sharipova commented on PR #14331:
URL: https://github.com/apache/lucene/pull/14331#issuecomment-2730506974

   @msokolov Thanks for the comment.
   I've experimented setting: beamCandidates0 to `M * 3` increasing it from the 
previous `M*2` when building merged graphs. 
   Graphs look better, but there are still significant speedups
   
   Evaluation is done with Luceneutil on these datasets:
   
   1. **quora-E5-small**; 522931 docs; 384 dims; 7 bits quantized; cosine metric
   
      - baseline: index time: **112.41s**,  force merge: **113.81s**
   
      - candidate: index time: **81.55s**, force merge: **70.87s**
   
   2. **cohere-wikipedia-v2**; 1M docs; 768 dims; 7 bits quantized; cosine 
metric
   
      - baseline: index time: **158.1s**, force merge: **425.20s**
   
      - candidate: index time: **122.95s**, force merge: **239.28s**
   
   3. **gist**; 960 dims, 1M docs; 7 bits quantized; euclidean metric
   
      - baseline: index time: **141.82s**, force merge: **536.07s**
   
      - candidate: index time: **119.26s**, force merge: **279.05s**
   
   4. **cohere-wikipedia-v3**; 1M docs; 1024 dims; 7 bits quantized; 
dot_product metric
   
      - baseline: index time: **211.86s**, force merge: **654.97s**
   
      - candidate: index time: **168.22s,** force merge: **414.12s**
   
   
![10_multiple](https://github.com/user-attachments/assets/3d78f854-386a-4ec7-b488-ad36ac1cdd58)
   
![10_single](https://github.com/user-attachments/assets/528bc255-e199-4764-9c02-12e96692394b)
   
![100_multiple](https://github.com/user-attachments/assets/7c45e8df-4c59-41c8-b922-f1c18274b8ae)
   
![100_single](https://github.com/user-attachments/assets/a0211c9e-2f8f-4e1f-9604-e5fe70c96eb5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org