mikemccand commented on issue #12358: URL: https://github.com/apache/lucene/issues/12358#issuecomment-1585584420
@uschindler I think the benchmark is quite fair at this point, from what I can tell -- we've been iterating [on that repo](https://github.com/Tony-X/search-benchmark-game) working to pull over luceneutil's enwiki corpus + tasks, ensuring the returned top N hits are very nearly identical, ensuring warmup + N iterations and take the fastest run is happening, measuring time on the server not the client, running both on a single segment index (makes results more comparable but not necessarily realistic to production cases), sprinkle 2% deletes on both indices, statically sorting both indices by the same `id` field, etc. It does indeed run a single JVM, but I an of the opposite opinion -- we need to run multiple JVM iterations to see how much hotspot noise is hurting. Darned java/hotspot coddling! The Rust results look quite consistent from one run to the next. It's also simple to run -- fork/clone the repo and follow the README steps. I'd love to see results from more diverse hardware than just my home beast boxes lol. > Still the mix of several languages makes it hard to understand. Ha! Java and Python and Rust! Start at [client.py](https://github.com/Tony-X/search-benchmark-game/blob/luceneutil/src/client.py) -- I think it's quite understandable if you launch off from there. More so than luceneutil!! > > When Tantitvy says it is 2 times faster then this is also unfair > > It's called tantivy. We do have a warmup phase. Oh -- sorry -- you mean no capitalization? So `tantivy` not `Tantivy`? I'll try to spell it correctly going forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
