iprithv opened a new pull request, #15978: URL: https://github.com/apache/lucene/pull/15978
## Description: Fixes #15967 The `printGraphBuildStatus` method in `HnswGraphBuilder.addVectors()` reports progress as `"built %d in %d/%d ms"` where the two times are meant to show incremental (since last print) and total (since start). However, during concurrent HNSW merging, each worker processes chunks of 2048 vectors, and the progress print fires every 10,000 nodes. This means the print fires **at most once per chunk**, always on the first matching node where `t == start`, producing identical values: ``` HNSW: built 950000 in 11447/11447 ms ← always duplicate ``` ## Changes **1. Suppress the broken in-loop progress for small ranges** Added a `numVectors > 10000` guard to skip the in-loop progress print for ranges ≤ 10,000 vectors. These ranges can trigger the print at most once, where `t == start` always produces duplicate incremental/total times. For large ranges (e.g., the non-concurrent `build()` path with millions of vectors), the in-loop progress continues to fire every 10K nodes with correct diverging times. **2. Add per-chunk completion logging** Every `addVectors()` call now emits a completion message with the range, vector count, and wall-clock elapsed time: ``` HNSW: addVectors [950000 952048): 2048 vectors in 47 ms ``` This enables deriving **effective concurrency** during HNSW merges by summing all chunk times that contributed to one merge and dividing by wall-clock elapsed time, as described in #15967. ## Output comparison **Before (concurrent merge chunk crossing a 10K boundary):** ``` HNSW: addVectors [950000 952048) HNSW: built 950000 in 11447/11447 ms ← duplicate times ``` **After:** ``` HNSW: addVectors [950000 952048): 2048 vectors in 11450 ms ``` **Before (non-concurrent build of 1M vectors):** ``` HNSW: build graph from 1000000 vectors HNSW: addVectors [0 1000000) HNSW: built 0 in 0/0 ms HNSW: built 10000 in 150/150 ms HNSW: built 20000 in 140/290 ms ... ``` **After:** ``` HNSW: build graph from 1000000 vectors HNSW: built 0 in 0/0 ms HNSW: built 10000 in 150/150 ms HNSW: built 20000 in 140/290 ms ... HNSW: addVectors [0 1000000): 1000000 vectors in 45000 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
