iprithv commented on code in PR #15978:
URL: https://github.com/apache/lucene/pull/15978#discussion_r3139839204
##########
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java:
##########
@@ -205,15 +205,27 @@ protected void addVectors(int minOrd, int maxOrd) throws
IOException {
throw new IllegalStateException("This HnswGraphBuilder is frozen and
cannot be updated");
}
long start = System.nanoTime(), t = start;
- if (infoStream.isEnabled(HNSW_COMPONENT)) {
- infoStream.message(HNSW_COMPONENT, "addVectors [" + minOrd + " " +
maxOrd + ")");
- }
+ int numVectors = maxOrd - minOrd;
for (int node = minOrd; node < maxOrd; node++) {
addGraphNode(node);
- if ((node % 10000 == 0) && infoStream.isEnabled(HNSW_COMPONENT)) {
+ // Skip in-loop progress for ranges <= 10000 where it would fire at most
once
+ // with identical incremental/total times (#15967).
+ if (numVectors > 10000 && (node % 10000 == 0) &&
infoStream.isEnabled(HNSW_COMPONENT)) {
t = printGraphBuildStatus(node, start, t);
}
}
+ if (infoStream.isEnabled(HNSW_COMPONENT)) {
+ long elapsed = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start);
+ infoStream.message(
Review Comment:
Sure! Added a shared AtomicLong cumulativeWorkTimeNS that each chunk
increments with its elapsed time. At the end of
HnswConcurrentMergeBuilder.build(), after all workers finish, it logs:
`merge completed: 100000 vectors, 12500.00 ms wall clock, 42300.45 ms
cumulative worker time, 3.38x effective concurrency`
This should make it straightforward to detect the single threaded merge
scenario, it would show ~1.0x effective concurrency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
