benwtrent commented on PR #13566: URL: https://github.com/apache/lucene/pull/13566#issuecomment-2226232962
I have benchmarked 2 data sets with 2 scenarios (int4 and int7 quantization) and have found no significant difference in runtime between this branch and the main branch. | | e5Small build | e5Small recall | e5small vectors visited | CohereV2 Build | CohereV2 recall | CohereV2 vectors visited | |----------------|---------------|----------------|-------------------------|----------------|-----------------|--------------------------| | candidate int7 | 582387 | 0.974 | 3090 | 1360523 | 0.825 | 4219 | | baseline int7 | 574839 | 0.974 | 3090 | 1360658 | 0.825 | 4218 | | candidate int4 | 571491 | 0.866 | 3118 | 1367738 | 0.516 | 4513 | | baseline int4 | 586543 | 0.866 | 3118 | 1381992 | 0.516 | 4512 | We should test a previously bad case to see how much longer indexing takes & how much recall is improved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
