orpiske commented on PR #11279: URL: https://github.com/apache/camel/pull/11279#issuecomment-1702954071
> > > You mean that we need to compare `925 ops/ms` for 4.1 with `922 ops/ms` for 4.0 in case of `SedaRoundTripTest.sendBlocking` and `229 ops/ms` for 4.1 with `242 ops/ms` for 4.0 in case of `SedaRoundTripTest.sendBlockingWithMultipleTypes`, right? > > > > > > Exactly. This tries to bring the code closer to what we had in 4.0, as I had to revert one of the performance improvements we had there. > > BTW, answering your other question: "Is there a real-life use case where we heavily create queues". > > The answer is yes, we do. Basically, that patch for 4.0 was breaking a scenario for an user that was heavily creating queues. > > If so if you check well the second case is better with Camel 4.0 and the first one is slightly better with Camel 4.1 but you have a higher Error `1.146` vs `0.462` which means that the results are more stable in Camel 4.0. > > I'm wondering if an improvement of less than 1 % is insignificant enough to be considered a real improvement as the results may vary a lot from one test to another. > > BTW, I've just realized that you [set forks to 1](https://github.com/apache/camel-performance-tests/blob/main/tests/camel-jmh/src/test/java/org/apache/camel/itest/jmh/SedaRoundTripTest.java#L55), maybe you should retry with 3 to get more reliable results? Normally that would be the case (i.e.: if we were comparing the results from the same test class). However, these tests are already running on forked executions: the iterations for different versions run separately (as in, the automation runs the tests for one version and then for another - not on the same JVM, I mean). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@camel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
