slow-J opened a new pull request, #16799: URL: https://github.com/apache/lucene/pull/16799
I was experimenting with native bulk scoring and I saw some good results in benchmarks with this setup. I am routing the scorer through VectorizationProvider so the native provider can score a batch of memory-mapped vectors with one int4DotProductSinglePackedBulk call. The C code scores four vectors per load of the query, with NEON dot product instructions on aarch64 Linux. Other platforms will still score one vector at a time. The scores are identical to scoring one vector at a time. The native code computes the same integer as `VectorUtil#int4DotProductSinglePacked`, and the corrections go through the same code. The query side doesn't change, and I think it could use the same call in a follow-up. ### Benchmarked via luceneutil `knnPerfTest.py` set up the same way I did in #16631: Cohere v3 Wikipedia 1024d, `ndoc=1_000_000`, `maxConn=64`, `beamWidthIndex=250`, `quantizeBits=4`, `forceMerge`, rerank off and at 32/8/4 bits, `numIndexThreads=8`, `numMergeThread=8`, `numMergeWorker=24`. Index only. 2 repetitions each of baseline + candidate. The candidate also has these options configured: `JAVA_EXE = "java -Dlucene.useNativeDotProduct=true -Djava.library.path=<dir>"` in `localconstants.py`, and the library built with GCC 10.5: ``` gcc -O3 -fPIC -shared -march=armv8-a -o <dir>/libdotProduct.so lucene/misc/src/c/dotProduct.c ``` Hardware: - EC2 Host: m6g.16xlarge - AWS Graviton2 (Neoverse N1), aarch64, no SVE, JDK 25.0.2 Build seconds (index + merge wait + force merge), averaged of 2 reps, main 82d73908cb -> this PR | rerank | total | Δ total | Δ force merge | |---|---:|---:|---:| | 32 bits | 1889.1 -> 1558.0 | -17.5% | -49.7% | | 8 bits | 923.1 -> 716.6 | -22.4% | -37.4% | | 4 bits | 1009.3 -> 630.5 | -37.5% | -17.0% | | no | 853.8 -> 594.9 | -30.3% | -59.6% | | 8 builds in total | 1168.8 -> 875.0 | -25.1% | -44.1% | Force merge and the total time are faster across all 8 baseline vs candidate runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
