LantaoJin commented on PR #16807: URL: https://github.com/apache/lucene/pull/16807#issuecomment-6121152556
> Got it - thanks for the explanation. I wonder what you are using for testing -- is it `luceneutil` or some package you cooked up? Maybe we can instrument luceneutil with such flags and scan logs for the optimization for failures as paet of its automation? Not `luceneutil`, but several standalone Java micro-benchmarks: An initial speed test covering all kernels revealed anomalies, and an end-to-end program which contains two `Lucene104HnswScalarQuantizedVectorsFormat` fields, one `UNSIGNED_BYTE` and one `SEVEN_BIT` in a single index, to compare kNN query latency. Individual kernel speeds were also measured using Lucene's built-in `benchmark-jmh`, but that tool failed to detect this issue. JMH forks a new JVM for every benchmark, `luceneutil`'s `knnPerfTest` launches a new JVM for each parameter combination, testing only one field and one encoding at a time. My thoughts are - Log scanning only reveals code actually executed under load. Therefore, `luceneutil` needs a mixed-scenario test—such as an index containing two quantization encodings simultaneously, or a byte vector field alongside an 8-bit quantized field. Logs may contain normal noise, such as transient intrinsic failures during warm-up or OSR (On-Stack Replacement) phases. Establishing a baseline before setting filter rules is necessary. Additionally, using `-XX:+LogCompilation` to write to a file is easier to parse than sending `PrintIntrinsics` output to `stdout`. - `VectorScorerFloat32Benchmark` already includes a `pollute` parameter that runs various scorers beforehand. A similar parameter could be added to the `VectorUtilBenchmark` byte kernel: running a kernel of a different type during the setup phase would allow JMH to detect the issue as a performance regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
