ilya-biryukov added a comment.
Here are some initial measurements with the current metrics that we have. We
seem to regress significantly in some cases. This is expected, since we only
check the prefix matches there.
@ioeric is working on including the non-prefix (more concretely, initialism
matches) into the metrics, we'll update when we get those metrics too.
==================================================================================================
OVERALL (excl. CROSS_NAMESPACE)
==================================================================================================
Total measurements: 109166 (+0)
Average latency (ms): 219.219589233 (-7)
All measurements:
MRR: 69.12 (-0.93) Top-1: 59.54% (-1.20%) Top-5: 81.21% (-0.48%)
Top-100: 96.06% (-0.10%)
Full identifiers:
MRR: 97.68 (-0.54) Top-1: 96.62% (-0.96%) Top-5: 98.96% (-0.07%)
Top-100: 99.15% (-0.01%)
Filter length 0-5:
MRR: 29.11 (+0.00) 62.02 (+0.41) 70.71 (-1.45)
73.41 (-1.18) 75.44 (-1.71) 78.78 (-2.50)
Top-1: 17.49% (+0.00%) 49.19% (+0.50%) 59.52%
(-1.51%) 62.47% (-1.41%) 65.27% (-2.11%) 69.25% (-3.53%)
Top-5: 42.45% (+0.00%) 78.99% (+0.39%) 85.48%
(-1.01%) 87.21% (-0.87%) 88.26% (-1.05%) 90.83% (-0.94%)
Top-100: 84.62% (+0.00%) 96.75% (+0.15%) 98.03%
(-0.16%) 98.22% (-0.23%) 98.29% (-0.31%) 98.47% (-0.23%)
==================================================================================================
DEFAULT
==================================================================================================
Total measurements: 52054 (+0)
Average latency (ms): 251.10333252 (-8)
All measurements:
MRR: 61.26 (-1.13) Top-1: 51.13% (-1.31%) Top-5: 74.10% (-0.68%)
Top-100: 92.60% (-0.21%)
Full identifiers:
MRR: 96.31 (-0.24) Top-1: 95.08% (-0.33%) Top-5: 97.93% (-0.09%)
Top-100: 98.30% (-0.01%)
Filter length 0-5:
MRR: 16.78 (+0.00) 48.89 (+0.32) 62.07 (-2.05)
66.09 (-1.67) 69.65 (-2.45) 72.49 (-2.23)
Top-1: 9.06% (+0.00%) 34.63% (+0.25%) 49.11%
(-2.03%) 53.07% (-1.99%) 58.01% (-2.83%) 61.95% (-2.74%)
Top-5: 23.57% (+0.00%) 68.10% (+0.52%) 79.78%
(-1.35%) 82.73% (-1.26%) 84.63% (-1.66%) 86.08% (-1.19%)
Top-100: 70.73% (+0.00%) 93.67% (+0.29%) 96.41%
(-0.31%) 96.77% (-0.47%) 96.92% (-0.63%) 97.07% (-0.47%)
==================================================================================================
EXPLICIT_MEMBER_ACCESS
==================================================================================================
Total measurements: 30470 (+0)
Average latency (ms): 114.196846008 (-5)
All measurements:
MRR: 67.95 (-1.05) Top-1: 58.04% (-1.53%) Top-5: 80.28% (-0.43%)
Top-100: 98.64% (-0.01%)
Full identifiers:
MRR: 98.19 (-1.45) Top-1: 96.69% (-2.75%) Top-5: 99.83% (-0.07%)
Top-100: 99.89% (+0.00%)
Filter length 0-5:
MRR: 27.70 (+0.00) 62.41 (+0.78) 68.45 (-1.11)
70.82 (-0.71) 72.00 (-1.10) 78.54 (-4.29)
Top-1: 16.41% (+0.00%) 49.21% (+1.05%) 57.13%
(-1.02%) 60.02% (-0.71%) 61.31% (-1.43%) 67.65% (-6.68%)
Top-5: 39.39% (+0.00%) 80.05% (+0.52%) 83.26%
(-1.18%) 84.73% (-0.69%) 85.38% (-0.69%) 92.24% (-1.05%)
Top-100: 94.25% (+0.00%) 99.21% (+0.02%) 99.20%
(-0.04%) 99.24% (-0.02%) 99.26% (-0.02%) 99.71% (+0.00%)
==================================================================================================
WANT_LOCAL
==================================================================================================
Total measurements: 26642 (+0)
Average latency (ms): 277.036773682 (-7)
All measurements:
MRR: 85.82 (-0.40) Top-1: 77.69% (-0.60%) Top-5: 96.18% (-0.13%)
Top-100: 99.88% (+0.00%)
Full identifiers:
MRR: 99.63 (-0.11) Top-1: 99.37% (-0.21%) Top-5: 99.91% (-0.02%)
Top-100: 99.93% (+0.00%)
Filter length 0-5:
MRR: 53.07 (+0.00) 86.94 (+0.19) 90.42 (-0.64)
91.19 (-0.74) 91.51 (-0.94) 92.59 (-0.81)
Top-1: 34.01% (+0.00%) 77.25% (+0.34%) 82.89%
(-1.01%) 84.30% (-1.06%) 84.99% (-1.45%) 86.95% (-1.23%)
Top-5: 80.13% (+0.00%) 98.80% (+0.00%) 99.32%
(-0.15%) 99.16% (-0.33%) 99.21% (-0.24%) 99.23% (-0.23%)
Top-100: 99.68% (+0.00%) 99.93% (+0.00%) 99.92%
(+0.00%) 99.92% (+0.00%) 99.91% (+0.00%) 99.90% (+0.00%)
==================================================================================================
CROSS_NAMESPACE
==================================================================================================
Total measurements: 18030 (+0)
Average latency (ms): 268.525360107 (-6)
All measurements:
MRR: 32.61 (-2.11) Top-1: 25.29% (-1.85%) Top-5: 40.29% (-2.63%)
Top-100: 72.80% (-3.47%)
Full identifiers:
MRR: 79.96 (-1.83) Top-1: 73.22% (-1.77%) Top-5: 88.01% (-2.14%)
Top-100: 98.71% (+0.00%)
Filter length 0-5:
MRR: 1.21 (+0.00) 11.40 (+0.28) 23.19 (-2.98)
29.33 (-3.42) 39.34 (-3.80) 46.81 (-3.58)
Top-1: 0.59% (+0.00%) 5.75% (+0.04%) 14.16%
(-2.45%) 19.95% (-2.53%) 29.11% (-3.62%) 36.17% (-3.05%)
Top-5: 1.44% (+0.00%) 16.38% (+0.41%) 31.15%
(-4.28%) 39.71% (-4.67%) 50.27% (-4.08%) 59.36% (-4.25%)
Top-100: 9.44% (+0.00%) 63.63% (+0.96%) 85.02%
(-2.19%) 82.54% (-8.21%) 86.24% (-8.50%) 88.93% (-7.92%)
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D59300/new/
https://reviews.llvm.org/D59300
_______________________________________________
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
