shatejas commented on PR #13985: URL: https://github.com/apache/lucene/pull/13985#issuecomment-2484035773
### Benchmarks #### Setup 1 - Opensearch cluster Ran with [opensearch benchmarks](https://github.com/opensearch-project/opensearch-benchmark) Total data nodes - 3 Total shards - 6 (2 per node), no replicas Memory - 128gb vCPU - 16 Dataset used: cohere-10m Baseline - OS 2.18 and lucene 9.12 Candidate - OS 2.16 and lucene [9.12 with readAdvice changes](https://github.com/apache/lucene/compare/branch_9_12...shatejas:lucene:branch_9_12) **Why was this tested with lucene 9.12?** Opensearch is not using lucene >9.12 for any of its version. Upgrading it to use lucene 10 requires significant changes. For candidate, required commits were cherry-picked Run 1: sequence of operations: delete-index -> create-index -> add documents -> force-merge -> search ##### Results | | Force-merge(ms) | Force-merge(hrs) | Search p50 | Search p90 | Search p99 | |-----------|-------------------|------------------|------------|------------|------------| | Baseline | 15795889.88313920 | 4hrs 23 mins | 9.6 | 10.8 | 14.7 | | Candidate | 15204143.95724240 | 4hrs 13mins | 10.7 | 12.0 | 15.0 | Run 2: Search performed on already indexed data from above run | | Search p50 | Search p90 | Search p99 | |-----------|------------|------------|------------| | Baseline | 9.7 | 10.6 | 12.1 | | Candidate | 10.4 | 11.3 | 12.5 | #### Setup 2: Used lucene-utils knnPerfTest.py Baseline - Lucene main Candidate - Lucene main with current commit **Baseline** | recall | latency (ms) | nDoc | topK | fanout | maxConn | beamWidth | quantized | index s | index docs/s | force merge s | num segments | index size (MB) | |-----------|------------|------------|------------|-----------|------------|------------|------------|-----------|------------|------------|------------|------------| | 0.644 | 0.428 | 50000 | 10 | 64 | 64 | 250 | no | 18.97 | 2635.18 | 1.89 | 1 | 20.62 | **Candidate** | recall | latency (ms) | nDoc | topK | fanout | maxConn | beamWidth | quantized | index s | index docs/s | force merge s | num segments | index size (MB) | |-----------|------------|------------|------------|-----------|------------|------------|------------|-----------|------------|------------|------------|------------| | 0.644 | 0.436 | 50000 | 10 | 64 | 64 | 250 | no | 20.20 | 2474.76 | 1.77 | 1 | 20.62 | There is a small affect on search latencies, its hard to say if its due to the change or just a fluctuation in the runs. I couldn't think of a reason that would of search latencies @jpountz @ChrisHegarty thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
