rmuir commented on pull request #177: URL: https://github.com/apache/lucene/pull/177#issuecomment-866272233
> Hmm, we added multiple JVMs long ago precisely because HotSpot was so unpredictable. I.e. we had clear examples where HotSpot would paint itself into a corner, compiling e.g. readVInt poorly and never re-compiling it, or something, such that no matter how long the benchmark ran, it would never reach as good performance as if you simply restarted the whole JVM and rolled the dice again. But maybe this situation has been improved and these were somehow early HotSpot bugs/issues and we could really remove multiple JVMs without harming how accurately we can extract the mean/variance performance of all our benchmark tasks? the JMH will do this too. I forget the defaults, but uses multiple jvm iterations and iterations within each jvm and warmup iterations. But it has smarts around the JIT compiler and can dump profiled assembly for its microbenchmarks. I never have noise issues with it. I think the issue is like "unit test" versus "integration test". The current big "integration test" (lucene util) is useful for some things: e.g. something has to tell us there is pollution from too many java abstractions going megamorphic and so on :) But I think it would be improved by providing some more diagnostics (LogCompilation or whatever, maybe JIT stats in the JFR output). Let it be a "canary" to find little ways to improve. But we have nothing setup to do simple noise-free microbenchmarks over some specific code, e.g. like "unit tests" running different query types. And for those you don't want crazy JFR and logging and stuff as it is so targeted, you can just dump the hot assembly code instead. For now if you want to do this, you are writing one-off stuff yourself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
