Re: [PR] stabilize vectorutil benchmark [lucene]

[via GitHub]

mikemccand commented on code in PR #12747:
URL: https://github.com/apache/lucene/pull/12747#discussion_r1379747014


##########
lucene/benchmark-jmh/src/java/org/apache/lucene/benchmark/jmh/VectorUtilBenchmark.java:
##########
@@ -24,8 +24,14 @@
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 @State(Scope.Benchmark)
-@Warmup(iterations = 3, time = 3)
-@Measurement(iterations = 5, time = 3)
+// first iteration is complete garbage, so make sure we really warmup
+@Warmup(iterations = 4, time = 1)
+// real iterations. not useful to spend tons of time here, better to fork more
+@Measurement(iterations = 5, time = 1)
+// engage some noise reduction
+@Fork(

Review Comment:
   Does `@Fork` mean spawn 3 JVMs, instead of running all of this in 1 JVM?



##########
lucene/benchmark-jmh/src/java/org/apache/lucene/benchmark/jmh/VectorUtilBenchmark.java:
##########
@@ -56,84 +62,72 @@ public void init() {
   }
 
   @Benchmark
-  @Fork(value = 1)
   public float binaryCosineScalar() {
     return VectorUtil.cosine(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(
-      value = 1,
-      jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
+  @Fork(jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
   public float binaryCosineVector() {
     return VectorUtil.cosine(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(value = 1)
   public int binaryDotProductScalar() {
     return VectorUtil.dotProduct(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(
-      value = 1,
-      jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
+  @Fork(jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
   public int binaryDotProductVector() {
     return VectorUtil.dotProduct(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(value = 1)
   public int binarySquareScalar() {
     return VectorUtil.squareDistance(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(
-      value = 1,

Review Comment:
   I guess the default for `value` is already `1`?



##########
lucene/benchmark-jmh/src/java/org/apache/lucene/benchmark/jmh/VectorUtilBenchmark.java:
##########
@@ -56,84 +62,72 @@ public void init() {
   }
 
   @Benchmark
-  @Fork(value = 1)
   public float binaryCosineScalar() {
     return VectorUtil.cosine(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(
-      value = 1,
-      jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
+  @Fork(jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
   public float binaryCosineVector() {
     return VectorUtil.cosine(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(value = 1)
   public int binaryDotProductScalar() {
     return VectorUtil.dotProduct(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(
-      value = 1,
-      jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
+  @Fork(jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
   public int binaryDotProductVector() {
     return VectorUtil.dotProduct(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(value = 1)
   public int binarySquareScalar() {
     return VectorUtil.squareDistance(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(
-      value = 1,
-      jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
+  @Fork(jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
   public int binarySquareVector() {
     return VectorUtil.squareDistance(bytesA, bytesB);
   }
 
   @Benchmark
-  @Fork(value = 1)
   public float floatCosineScalar() {
     return VectorUtil.cosine(floatsA, floatsB);
   }
 
   @Benchmark
   @Fork(
-      value = 1,
+      value = 15,
       jvmArgsPrepend = {"--add-modules=jdk.incubator.vector"})
   public float floatCosineVector() {
     return VectorUtil.cosine(floatsA, floatsB);
   }
 
   @Benchmark
-  @Fork(value = 1)
   public float floatDotProductScalar() {
     return VectorUtil.dotProduct(floatsA, floatsB);
   }
 
   @Benchmark
   @Fork(
-      value = 1,
+      value = 15,

Review Comment:
   Curious that the vector tasks seem to need more forking / are more noisy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org