aylonsk commented on PR #14932: URL: https://github.com/apache/lucene/pull/14932#issuecomment-3059012712
Thanks for your response! My apologies, I forgot to post my results from LuceneUtil. Because I noticed variance between each run, I decided to test each set of hyperparameters 10 times and take the median for latency, netCPU, and AvgCpuCount. Therefore, my results aren't in the standard table format. I ran 12 comparison tests in total, each a different combination of HPs. Here were the variables I kept the same: (topK=100, fanout=50, beamWidth=250, numSegments=1) Here are some specific tests: BENCHMARKS (10 runs per test): 1. Base HP’s: nDocs=500,000, maxConn=64, quantized=no, numSegments=1 Baseline: Recall: 0.832 Latency (Median): 0.73 ms NetCPU (Median) 0.708 ms AvgCPUCount (Median): 0.973 ms Index Size: 220.55MB Vec Disk/Vec RAM: 190.735MB Candidate: Recall: 0.835 Latency (Median): 0.7 ms NetCPU (Median) 0.677 ms AvgCPUCount (Median): 0.966 ms Index Size: 220.12MB Vec Disk/Vec RAM: 190.735MB Latency Improvement: ~4.11% speedup 2. nDocs=500,000, maxConn=32, quantized=no, numSegments=1 Baseline: Recall: 0.834 Latency (Median): 0.722 ms NetCPU (Median): 0.701 ms AvgCPUCount (Median): 0.966 ms Index Size: 220.19MB Vec Disk/Vec RAM: 190.735MB Candidate: Recall: 0.83 Latency (Median): 0.691 ms NetCPU (Median): 0.665 ms AvgCPUCount (Median): 0.96 ms Index Size: 219.67MB Vec Disk/Vec RAM: 190.735MB Latency Improvement: ~4.3% speedup 8. nDocs=500,000, maxConn=32, quantized=7bits, numSegments=1 Baseline: Recall: 0.671 Latency (Median): 1.2935 ms NetCPU (Median): 1.2635 ms AvgCpuCount (Median): 0.976 ms Index Size: 255.74 ms Vec Disk: 240.326MB Vec RAM: 49.591MB Candidate: Recall: 0.696 Latency (Median): 1.2525 ms NetCPU (Median): 1.192 ms AvgCPUCount (Median): 0.974 ms Index Size: 259.34MB Vec Disk: 240.326MB Vec RAM: 49.591MB Latency Improvement: ~3.17% speedup 10. nDocs=2,000,000, maxConn=32, quantized=7bits, numSegments=1 Baseline: Recall: 0.74 Latency (Median): 2.6675 ms NetCPU (Median): 2.545 ms AvgCpuCount (Median): 0.969 ms Index Size: 1049.52MB Vec Disk: 961.30MB Vec RAM: 198.364MB Candidate: Recall: 0.717 Latency (Median): 2.521 ms NetCPU (Median): 2.398 ms AvgCPUCount (Median): 0.98 ms Index Size: 1043.27MB Vec Disk: 961.304MB Vec RAM: 198.364MB Latency Improvement: 5.49% speedup 11. nDocs=100,000, maxConn=64, quantized=7bits, numSegments=1 Baseline: Recall: 0.848 Latency (Median): 2.305 NetCPU (Median): 2.2575 AvgCpuCount (Median): 0.976 Index Size: 51.52MB Vec Disk: 48.07MB Vec RAM: 9.918MB Candidate: Recall: 0.848 Latency (Median): 1.85 ms NetCPU (Median): 1.80 ms AvgCPUCount (Median): 0.974 ms Index Size: 51.52MB Vec Disk: 48.07MB Vec RAM: 9.918MB Latency Improvement: ~18.1% speedup While the degree of improvement varied between tests, all tests except 1 showed improvement in latency over the baseline. Considering how simple and non-intrusive this implementation is, I think it would be an easy net benefit. Thank you for letting me know about the backwards compatibility requirement. I will look into fixing that tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
