zacharymorn commented on pull request #101:
URL: https://github.com/apache/lucene/pull/101#issuecomment-840255508
I'm able to run wikibigall with the above configuration (and a few small
changes to `localrun.py` to take in vector dictionary and file), and get the
following results:
wikibigall run 1
```
TaskQPS baseline StdDevQPS my_modified_version
StdDev Pct diff p-value
AndMedOrHighHigh 25.47 (2.5%) 21.21 (4.1%)
-16.7% ( -22% - -10%) 0.000
OrHighMed 72.05 (2.5%) 60.17 (4.1%)
-16.5% ( -22% - -10%) 0.000
Fuzzy2 31.51 (8.4%) 29.27 (9.5%)
-7.1% ( -23% - 11%) 0.012
Fuzzy1 59.00 (8.4%) 56.38 (9.8%)
-4.4% ( -20% - 14%) 0.122
TermDayOfYearSort 48.68 (14.6%) 46.86 (8.9%)
-3.7% ( -23% - 23%) 0.328
TermMonthSort 61.44 (9.3%) 60.23 (9.8%)
-2.0% ( -19% - 18%) 0.514
TermTitleSort 110.13 (8.9%) 108.54 (9.2%)
-1.4% ( -18% - 18%) 0.616
TermDTSort 56.74 (9.5%) 56.04 (10.2%)
-1.2% ( -19% - 20%) 0.692
TermGroup10K 15.95 (2.6%) 15.75 (3.2%)
-1.2% ( -6% - 4%) 0.183
TermGroup100 19.34 (3.5%) 19.11 (3.4%)
-1.2% ( -7% - 5%) 0.276
TermBGroup1M 17.92 (2.7%) 17.72 (3.8%)
-1.1% ( -7% - 5%) 0.280
TermGroup1M 19.06 (2.5%) 18.86 (3.1%)
-1.1% ( -6% - 4%) 0.234
TermBGroup1M1P 43.93 (3.8%) 43.66 (4.4%)
-0.6% ( -8% - 7%) 0.633
SpanNear 24.55 (2.5%) 24.46 (2.4%)
-0.4% ( -5% - 4%) 0.620
SloppyPhrase 2.99 (8.2%) 2.98 (8.0%)
-0.3% ( -15% - 17%) 0.897
IntNRQ 139.92 (1.2%) 139.61 (1.7%)
-0.2% ( -3% - 2%) 0.627
IntervalsOrdered 2.02 (3.7%) 2.02 (3.6%)
-0.1% ( -7% - 7%) 0.957
BrowseMonthSSDVFacets 19.28 (1.0%) 19.27 (0.9%)
-0.0% ( -1% - 1%) 0.888
BrowseDayOfYearSSDVFacets 17.61 (1.9%) 17.61 (1.8%)
0.0% ( -3% - 3%) 0.989
Respell 48.40 (2.6%) 48.59 (3.1%)
0.4% ( -5% - 6%) 0.660
PKLookup 211.17 (3.4%) 212.28 (3.6%)
0.5% ( -6% - 7%) 0.631
BrowseDayOfYearTaxoFacets 7.30 (6.4%) 7.34 (5.8%)
0.6% ( -10% - 13%) 0.766
Prefix3 120.15 (6.1%) 120.95 (4.6%)
0.7% ( -9% - 12%) 0.694
Phrase 54.76 (3.6%) 55.13 (4.3%)
0.7% ( -6% - 8%) 0.586
BrowseDateTaxoFacets 7.62 (6.7%) 7.67 (6.1%)
0.7% ( -11% - 14%) 0.718
Wildcard 72.89 (2.8%) 73.46 (2.2%)
0.8% ( -4% - 5%) 0.331
BrowseMonthTaxoFacets 8.42 (7.0%) 8.49 (6.4%)
0.8% ( -11% - 15%) 0.694
TermDateFacets 8.56 (7.4%) 8.64 (7.3%)
0.9% ( -12% - 16%) 0.702
VectorSearch 954.06 (3.7%) 967.28 (2.5%)
1.4% ( -4% - 7%) 0.167
AndHighHigh 32.04 (3.1%) 32.65 (4.1%)
1.9% ( -5% - 9%) 0.100
AndHighMed 83.72 (1.8%) 85.38 (2.9%)
2.0% ( -2% - 6%) 0.009
AndHighOrMedMed 44.41 (2.2%) 45.34 (4.2%)
2.1% ( -4% - 8%) 0.049
Term 1132.83 (5.2%) 1157.74 (5.0%)
2.2% ( -7% - 13%) 0.173
OrHighHigh 11.35 (2.9%) 16.95 (4.1%)
49.4% ( 41% - 58%) 0.000
```
wikibigall run 2
```
TaskQPS baseline StdDevQPS my_modified_version
StdDev Pct diff p-value
TermDayOfYearSort 59.31 (11.3%) 56.25 (9.5%)
-5.2% ( -23% - 17%) 0.116
TermTitleSort 175.57 (10.3%) 169.31 (9.3%)
-3.6% ( -21% - 17%) 0.252
TermMonthSort 61.73 (10.5%) 59.73 (9.2%)
-3.2% ( -20% - 18%) 0.300
TermBGroup1M 11.87 (3.2%) 11.57 (2.6%)
-2.6% ( -8% - 3%) 0.005
TermGroup10K 15.60 (2.8%) 15.26 (3.3%)
-2.2% ( -8% - 4%) 0.023
TermGroup100 23.39 (2.6%) 22.94 (3.0%)
-1.9% ( -7% - 3%) 0.030
TermGroup1M 12.16 (2.6%) 11.94 (2.9%)
-1.8% ( -7% - 3%) 0.036
TermDTSort 78.08 (6.4%) 76.75 (5.4%)
-1.7% ( -12% - 10%) 0.361
Fuzzy1 28.49 (5.6%) 28.08 (5.8%)
-1.4% ( -12% - 10%) 0.419
TermBGroup1M1P 42.98 (2.9%) 42.49 (3.3%)
-1.1% ( -7% - 5%) 0.247
IntNRQ 136.00 (3.4%) 134.92 (3.5%)
-0.8% ( -7% - 6%) 0.473
AndHighHigh 22.60 (3.9%) 22.42 (2.4%)
-0.8% ( -6% - 5%) 0.455
TermDateFacets 8.38 (7.0%) 8.32 (6.6%)
-0.7% ( -13% - 13%) 0.738
Prefix3 33.76 (3.1%) 33.55 (2.8%)
-0.6% ( -6% - 5%) 0.502
Wildcard 88.25 (3.0%) 87.72 (2.3%)
-0.6% ( -5% - 4%) 0.479
BrowseDayOfYearSSDVFacets 17.39 (1.4%) 17.31 (1.5%)
-0.4% ( -3% - 2%) 0.363
Term 1144.71 (8.2%) 1139.98 (8.5%)
-0.4% ( -15% - 17%) 0.876
VectorSearch 807.48 (4.7%) 804.27 (4.9%)
-0.4% ( -9% - 9%) 0.794
IntervalsOrdered 0.82 (5.3%) 0.82 (5.4%)
-0.3% ( -10% - 11%) 0.872
PKLookup 207.38 (4.8%) 206.99 (5.5%)
-0.2% ( -10% - 10%) 0.910
Respell 48.27 (3.6%) 48.18 (3.6%)
-0.2% ( -7% - 7%) 0.878
BrowseDateTaxoFacets 7.58 (6.8%) 7.57 (6.5%)
-0.2% ( -12% - 14%) 0.933
BrowseDayOfYearTaxoFacets 7.26 (6.3%) 7.24 (6.2%)
-0.2% ( -11% - 13%) 0.932
SloppyPhrase 5.55 (5.7%) 5.54 (5.8%)
-0.1% ( -10% - 12%) 0.946
BrowseMonthTaxoFacets 8.38 (6.7%) 8.38 (6.6%)
-0.1% ( -12% - 14%) 0.969
SpanNear 2.12 (3.5%) 2.12 (3.9%)
0.0% ( -7% - 7%) 0.993
BrowseMonthSSDVFacets 19.52 (5.7%) 19.53 (5.7%)
0.0% ( -10% - 12%) 0.985
Phrase 59.48 (3.8%) 59.72 (3.9%)
0.4% ( -6% - 8%) 0.735
AndHighMed 60.30 (4.6%) 60.57 (2.3%)
0.4% ( -6% - 7%) 0.705
Fuzzy2 53.28 (11.3%) 53.93 (10.6%)
1.2% ( -18% - 26%) 0.727
AndMedOrHighHigh 27.58 (3.1%) 27.93 (3.4%)
1.3% ( -5% - 8%) 0.209
AndHighOrMedMed 24.38 (3.1%) 25.06 (3.3%)
2.8% ( -3% - 9%) 0.006
OrHighMed 45.08 (4.3%) 57.44 (6.9%)
27.4% ( 15% - 40%) 0.000
OrHighHigh 11.22 (4.8%) 16.16 (9.8%)
44.0% ( 28% - 61%) 0.000
```
wikibigall run 3
```
TaskQPS baseline StdDevQPS my_modified_version
StdDev Pct diff p-value
Fuzzy2 51.26 (10.6%) 47.91 (11.4%)
-6.5% ( -25% - 17%) 0.060
TermDTSort 83.68 (11.6%) 79.69 (7.7%)
-4.8% ( -21% - 16%) 0.126
Fuzzy1 60.65 (6.6%) 57.96 (8.5%)
-4.4% ( -18% - 11%) 0.065
TermMonthSort 61.55 (15.5%) 59.70 (8.4%)
-3.0% ( -23% - 24%) 0.447
TermTitleSort 88.13 (15.1%) 85.68 (8.4%)
-2.8% ( -22% - 24%) 0.471
TermDayOfYearSort 47.32 (8.7%) 46.58 (10.0%)
-1.5% ( -18% - 18%) 0.602
Wildcard 37.71 (9.9%) 37.32 (9.1%)
-1.0% ( -18% - 19%) 0.729
Prefix3 164.10 (12.8%) 162.91 (11.5%)
-0.7% ( -22% - 27%) 0.850
TermGroup100 23.62 (3.0%) 23.50 (2.9%)
-0.5% ( -6% - 5%) 0.617
TermBGroup1M 11.90 (2.7%) 11.86 (3.1%)
-0.3% ( -5% - 5%) 0.714
Term 1044.58 (8.0%) 1041.84 (6.5%)
-0.3% ( -13% - 15%) 0.910
TermGroup10K 12.18 (2.9%) 12.15 (2.7%)
-0.2% ( -5% - 5%) 0.808
IntervalsOrdered 3.91 (2.7%) 3.91 (2.7%)
-0.1% ( -5% - 5%) 0.934
BrowseDayOfYearSSDVFacets 17.47 (2.3%) 17.47 (2.0%)
-0.0% ( -4% - 4%) 0.994
VectorSearch 851.38 (5.2%) 851.53 (5.6%)
0.0% ( -10% - 11%) 0.992
SpanNear 24.59 (2.2%) 24.61 (2.3%)
0.1% ( -4% - 4%) 0.923
TermGroup1M 15.44 (2.7%) 15.45 (2.9%)
0.1% ( -5% - 5%) 0.936
BrowseMonthSSDVFacets 19.19 (1.4%) 19.22 (1.5%)
0.2% ( -2% - 3%) 0.746
TermBGroup1M1P 18.19 (4.7%) 18.24 (4.6%)
0.3% ( -8% - 9%) 0.858
IntNRQ 136.29 (3.4%) 136.75 (3.5%)
0.3% ( -6% - 7%) 0.755
Respell 40.87 (5.4%) 41.04 (4.8%)
0.4% ( -9% - 11%) 0.793
Phrase 24.71 (3.1%) 24.87 (2.7%)
0.7% ( -4% - 6%) 0.472
PKLookup 205.99 (5.9%) 208.11 (6.4%)
1.0% ( -10% - 14%) 0.596
BrowseDayOfYearTaxoFacets 7.04 (7.8%) 7.11 (7.5%)
1.0% ( -13% - 17%) 0.665
BrowseDateTaxoFacets 7.36 (7.9%) 7.45 (7.7%)
1.1% ( -13% - 18%) 0.646
SloppyPhrase 1.17 (10.0%) 1.19 (9.8%)
1.2% ( -16% - 23%) 0.705
AndHighMed 45.25 (3.5%) 45.82 (3.8%)
1.3% ( -5% - 8%) 0.274
BrowseMonthTaxoFacets 8.13 (7.9%) 8.25 (7.9%)
1.4% ( -13% - 18%) 0.569
AndHighHigh 23.27 (3.6%) 23.60 (4.0%)
1.4% ( -5% - 9%) 0.227
TermDateFacets 11.08 (9.9%) 11.27 (10.0%)
1.7% ( -16% - 23%) 0.582
AndHighOrMedMed 24.91 (3.8%) 25.72 (4.0%)
3.3% ( -4% - 11%) 0.008
AndMedOrHighHigh 10.00 (4.2%) 10.61 (4.2%)
6.1% ( -2% - 15%) 0.000
OrHighMed 59.17 (5.5%) 67.57 (4.5%)
14.2% ( 3% - 25%) 0.000
OrHighHigh 17.06 (4.9%) 24.26 (5.2%)
42.2% ( 30% - 55%) 0.000
```
I'll run it for the other two implementations as well and post the results
in the other PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
