kaivalnp commented on PR #15979:
URL: https://github.com/apache/lucene/pull/15979#issuecomment-4988199136
### Benchmarks
#### Single-threaded indexing
Common
```
Results:
NOTE: nDoc = 100000 for all runs; skipping column
NOTE: topK = 100 for all runs; skipping column
NOTE: fanout = 100 for all runs; skipping column
NOTE: maxConn = 64 for all runs; skipping column
NOTE: beamWidth = 250 for all runs; skipping column
NOTE: quantized = no for all runs; skipping column
NOTE: num_segments = 1 for all runs; skipping column
NOTE: filterSelectivity = 0.50 for all runs; skipping column
```
`main`
```
visited index(s) index_docs/s index_size(MB) filterStrategy
recall latency(ms) netCPU avgCpuCount
6184 520.32 192.19 603.01 index-time-filter
0.995 2.069 2.068 0.999
12877 362.57 275.81 402.98 query-time-pre-filter
0.989 4.417 4.416 1.000
```
This PR
```
visited index(s) index_docs/s index_size(MB) filterStrategy
recall latency(ms) netCPU
6184 531.66 188.09 408.98 index-time-filter
0.995 2.174 2.173
12877 359.93 277.83 403.36 query-time-pre-filter
0.989 4.478 4.477
```
#### Multi-threaded indexing with force-merge
(same Common values)
`main`
```
visited index(s) index_docs/s merge(s) force_merge(s) index_size(MB)
filterStrategy recall latency(ms) netCPU
6196 22.27 4490.14 21.52 36.74 603.02
index-time-filter 0.995 1.942 1.941
12944 18.91 5289.61 19.02 19.52 403.01
query-time-pre-filter 0.989 4.343 4.343
```
This PR
```
visited index(s) index_docs/s force_merge(s) index_size(MB)
filterStrategy recall latency(ms) netCPU
6199 38.90 2570.83 44.97 409.01
index-time-filter 0.995 2.034 2.033
12883 29.90 3344.71 31.78 403.37
query-time-pre-filter 0.989 4.352 4.351
```
#### Notes
- The **Single-threaded indexing** run is slightly faster for the
de-duplicating format in case of no explicit duplicates, and <3% slower with
50% explicit duplicates (i.e. a second field having a subset of 50% docs of the
main one), which appears to be in the range of HNSW noise.
- The **Multi-threaded indexing with force-merge** run is not directly
comparable, because the auto-started merges were different (as @mikemccand
mentioned). That said, the _sum_ of `index(s)` + `merge(s)` + `force_merge(s)`
is \~4% slower for no explicit duplicates, and \~7% slower with 50% explicit
duplicates.
- The query latency was consistently slower for the de-duplicating vector
format across multiple runs, but this was <5% slower. One point to note: the
`ordToVecOrd` indirection (applicable before scoring _each_ vector) is
currently off-heap, and can likely be sped-up by maintaining an on-heap `int[]`
map. I'm hoping that the memory residency of the off-heap map will be in-line
with the HNSW graph.
- Note the drop in `index_size(MB)` -- the index size is about the same with
50% explicit duplicates and without them (the difference being the second HNSW
graph).
- These benchmarks also highlight the importance of indexing a separate
field for filters known at indexing time (the `index-time-filter` option is >2x
faster than `query-time-pre-filter`, not even including overheads of `BitSet`
creation and maintenance).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
