mikemccand commented on pull request #177: URL: https://github.com/apache/lucene/pull/177#issuecomment-866784477
> > > The problem with luceneutil is also that it respawns a JVM multiple times. > > > > > > Hmm, we added multiple JVMs long ago precisely because HotSpot was so unpredictable. I.e. we had clear examples where HotSpot would paint itself into a corner, compiling e.g. `readVInt` poorly and never re-compiling it, or something, such that no matter how long the benchmark ran, it would never reach as good performance as if you simply restarted the whole JVM and rolled the dice again. But maybe this situation has been improved and these were somehow early HotSpot bugs/issues and we could really remove multiple JVMs without harming how accurately we can extract the mean/variance performance of all our benchmark tasks? > > This is also not reality: Would you restart your Elasticsearch server from time to time because you think there might be a broken `readVInt()` optimization? Yeah, that is true! But perhaps it shouldn't be the case :) Maybe Elasticsearch/OpenSearch/Solr should spawn JVM a few times until they get a "good" `readVInt` compilation! The noisy mis-compilation was such a sizable impact (back then, hopefully not anymore?). If we only ran benchmarks in nightly runs so that we could see that noise/variance with time, maybe we could do just one JVM. But when a developer is trying to test an exciting optimization, in the privacy of their `git clone`, it really sucks to have hotspot noise completely drown out any small gains your optimization might show! Benchmarking is hard :) > Here are the berlinbuzzwords slides about this: https://2021.berlinbuzzwords.de/sites/berlinbuzzwords.de/files/2021-06/The%20future%20of%20Lucene%27s%20MMapDirectory.pdf Oooh, thanks for sharing! The talk looks AWESOME! I will watch recording when it's out :) You should share these slides on Twitter too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
