Pranshu-S commented on PR #16710:
URL: https://github.com/apache/lucene/pull/16710#issuecomment-5948331999
Runs against different quantisation levels:
```
============================================================================
>>> quantizeBits=1
============================================================================
=================================================================================
COMPARISON (candidate=lucene_candidate+dedup vs
baseline=lucene_baseline+non-dedup;
delta/pct = candidate - baseline)
metric baseline/non-dedup(n=5) candidate/dedup(n=5) delta
pct
----------------- ----------------------- -------------------- -------
------
recall 0.714±0.002 0.679±0.003 -0.035
-4.9%
latency(ms) 0.437±0.007 0.444±0.007 +0.007
+1.6%
netCPU 0.429±0.007 0.435±0.006 +0.006
+1.4%
avgCpuCount 0.982±0.003 0.981±0.003 -0.001
-0.1%
nDoc 10000 10000
searchType KNN KNN
topK 10 10
fanout 100 100
resultSimilarity N/A N/A
decay N/A N/A
resultCount 10.000 10.000
maxConn 32 32
beamWidth 200 200
quantized 1 bits 1 bits
indexEncoding float16 float16
visited 2364.000±5.292 2366.400±14.673 +2.400
+0.1%
index(s) 6.194±0.701 5.926±0.527 -0.268
-4.3%
index_docs/s 1631.810±192.762 1698.786±163.755 +66.976
+4.1%
merge(s) 0.000±0.000 0.000±0.000 +0.000
force_merge(s) 9.184±0.161 8.912±0.053 -0.272
-3.0%
num_segments 1.000±0.000 1.000±0.000 +0.000
+0.0%
index_size(MB) 32.418±0.084 21.906±0.005 -10.512
-32.4%
filterStrategy index-time-filter index-time-filter
filterSelectivity 0.50 0.50
overSample 1.000 1.000
vec_disk(MB) 20.885±0.000 20.885±0.000 +0.000
+0.0%
vec_RAM(MB) 1.354±0.000 1.354±0.000 +0.000
+0.0%
bp-reorder false false
indexType HNSW HNSW
rerank no no
```
```
============================================================================
>>> quantizeBits=2
============================================================================
=================================================================================
COMPARISON (candidate=lucene_candidate+dedup vs
baseline=lucene_baseline+non-dedup;
delta/pct = candidate - baseline)
metric baseline/non-dedup(n=5) candidate/dedup(n=5) delta
pct
----------------- ----------------------- -------------------- -------
------
recall 0.834±0.005 0.800±0.001 -0.034
-4.1%
latency(ms) 0.517±0.018 0.530±0.004 +0.013
+2.6%
netCPU 0.510±0.018 0.522±0.004 +0.012
+2.3%
avgCpuCount 0.986±0.001 0.984±0.003 -0.002
-0.2%
nDoc 10000 10000
searchType KNN KNN
topK 10 10
fanout 100 100
resultSimilarity N/A N/A
decay N/A N/A
resultCount 10.000 10.000
maxConn 32 32
beamWidth 200 200
quantized 2 bits 2 bits
indexEncoding float16 float16
visited 2161.200±49.226 2197.000±11.424 +35.800
+1.7%
index(s) 6.472±0.683 6.346±0.244 -0.126
-1.9%
index_docs/s 1560.354±171.079 1578.426±61.459 +18.072
+1.2%
merge(s) 0.000±0.000 0.000±0.000 +0.000
force_merge(s) 11.300±0.352 10.272±0.261 -1.028
-9.1%
num_segments 1.000±0.000 1.000±0.000 +0.000
+0.0%
index_size(MB) 34.144±0.171 23.080±0.000 -11.064
-32.4%
filterStrategy index-time-filter index-time-filter
filterSelectivity 0.50 0.50
overSample 1.000 1.000
vec_disk(MB) 22.106±0.000 22.106±0.000 +0.000
+0.0%
vec_RAM(MB) 2.575±0.000 2.575±0.000 +0.000
+0.0%
bp-reorder false false
indexType HNSW HNSW
rerank no no
```
```
============================================================================
>>> quantizeBits=4
============================================================================
==================================================================================
COMPARISON (candidate=lucene_candidate+dedup vs
baseline=lucene_baseline+non-dedup;
delta/pct = candidate - baseline)
metric baseline/non-dedup(n=5) candidate/dedup(n=5) delta
pct
----------------- ----------------------- -------------------- --------
------
recall 0.922±0.001 0.913±0.002 -0.009
-1.0%
latency(ms) 0.553±0.016 0.547±0.018 -0.006
-1.0%
netCPU 0.546±0.015 0.541±0.018 -0.005
-1.0%
avgCpuCount 0.987±0.002 0.988±0.002 +0.001
+0.1%
nDoc 10000 10000
searchType KNN KNN
topK 10 10
fanout 100 100
resultSimilarity N/A N/A
decay N/A N/A
resultCount 10.000 10.000
maxConn 32 32
beamWidth 200 200
quantized 4 bits 4 bits
indexEncoding float16 float16
visited 2121.800±24.201 2116.800±39.996 -5.000
-0.2%
index(s) 6.456±0.441 5.446±0.629 -1.010
-15.6%
index_docs/s 1555.260±114.384 1854.910±205.027 +299.650
+19.3%
merge(s) 0.000±0.000 0.000±0.000 +0.000
force_merge(s) 11.134±0.199 9.996±0.321 -1.138
-10.2%
num_segments 1.000±0.000 1.000±0.000 +0.000
+0.0%
index_size(MB) 37.900±0.104 25.520±0.000 -12.380
-32.7%
filterStrategy index-time-filter index-time-filter
filterSelectivity 0.50 0.50
overSample 1.000 1.000
vec_disk(MB) 24.567±0.000 24.567±0.000 +0.000
+0.0%
vec_RAM(MB) 5.035±0.000 5.035±0.000 +0.000
+0.0%
bp-reorder false false
indexType HNSW HNSW
rerank no no
```
```
============================================================================
>>> quantizeBits=7
============================================================================
==================================================================================
COMPARISON (candidate=lucene_candidate+dedup vs
baseline=lucene_baseline+non-dedup;
delta/pct = candidate - baseline)
metric baseline/non-dedup(n=5) candidate/dedup(n=5) delta
pct
----------------- ----------------------- -------------------- --------
------
recall 0.978±0.002 0.976±0.001 -0.002
-0.2%
latency(ms) 0.559±0.014 0.583±0.015 +0.023
+4.2%
netCPU 0.553±0.014 0.577±0.015 +0.024
+4.3%
avgCpuCount 0.988±0.001 0.989±0.001 +0.001
+0.1%
nDoc 10000 10000
searchType KNN KNN
topK 10 10
fanout 100 100
resultSimilarity N/A N/A
decay N/A N/A
resultCount 10.000 10.000
maxConn 32 32
beamWidth 200 200
quantized 7 bits 7 bits
indexEncoding float16 float16
visited 2118.800±12.755 2115.600±12.482 -3.200
-0.2%
index(s) 6.036±0.620 5.590±0.622 -0.446
-7.4%
index_docs/s 1670.912±165.291 1806.012±193.513 +135.100
+8.1%
merge(s) 0.000±0.000 0.000±0.000 +0.000
force_merge(s) 11.210±0.195 10.080±0.100 -1.130
-10.1%
num_segments 1.000±0.000 1.000±0.000 +0.000
+0.0%
index_size(MB) 45.096±0.212 30.398±0.008 -14.698
-32.6%
filterStrategy index-time-filter index-time-filter
filterSelectivity 0.50 0.50
overSample 1.000 1.000
vec_disk(MB) 29.449±0.000 29.449±0.000 +0.000
+0.0%
vec_RAM(MB) 9.918±0.000 9.918±0.000 +0.000
+0.0%
bp-reorder false false
indexType HNSW HNSW
rerank no no
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
