On 24/02/2016 16:20, Walter Underwood wrote:
Click through rate (CTR) is fundamental. That is easy to understand
and integrates well with other business metrics like conversion. CTR
is at least one click anywhere in the result set (first page, second
page, …). Count multiple clicks as a single success. The metric is,
“at least one click”.
I saw an interesting talk last year by Susan Dumais of Microsoft
Research which contained the surprising (to me) statistic that clicks
only predict relevance 45% of the time: here's the talk and the original
paper:
www.iskouk.org/sites/default/files/Dumais_Slides2015-11-06.PDF
http://research.microsoft.com/en-us/um/people/sdumais/TOIS-p147-fox.pdf
One also shouldn't forget manual testing of relevance, something that we
and Open Source Connections are working on a lot at present. Content
owners / business types are far better at judging relevance than
developers who may not understand the rationale. I talked about this at
the British Computer Society last year:
http://www.flax.co.uk/blog/2015/11/27/search-solutions-2015-towards-new-model-search-relevance-testing/
and here's the tool OSC has developed that we're using: www.quepid.com
Cheers
Charlie
No hit rate is sort of useful, but you need to know which queries are
getting no hits, so you can fix it.
For latency metrics, look at 90th percentile or 95th percentile.
Average is useless because response time is a one-sided distribution,
so it will be thrown off by outliers. Percentiles have a direct
customer satisfaction interpretation. 90% of searches were under one
second, for example. Median response time should be very, very fast
because of caching in Solr. During busy periods, our median response
time is about 1.5 ms.
Number of different queries per conversion is a good way to look how
query assistance is working. Things like autosuggest, fuzzy, etc.
About 10% of queries will be misspelled, so you do need to deal with
that.
Finding underperforming queries is trickier. I really need to write
an article on that.
“Search Analytics for Your Site” by Lou Rosenfeld is a good
introduction.
http://rosenfeldmedia.com/books/search-analytics-for-your-site/
<http://rosenfeldmedia.com/books/search-analytics-for-your-site/>
Sea Urchin is doing some good work in search metrics:
https://seaurchin.io/ <https://seaurchin.io/>
wunder Walter Underwood wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog) Search Guy, Chegg
On Feb 24, 2016, at 2:38 AM, Emir Arnautovic
<emir.arnauto...@sematext.com> wrote:
Hi Bill, You can take a look at Sematext's search analytics
(https://sematext.com/search-analytics). It provides some of
metrics you mentioned, plus some additional (top queries, CTR,
click stats, paging stats etc.). In combination with Sematext's
performance metrics (https://sematext.com/spm) you can have full
picture of your search infrastructure.
Regards, Emir
-- Monitoring * Alerting * Anomaly Detection * Centralized Log
Management Solr & Elasticsearch Support * http://sematext.com/
On 24.02.2016 04:07, William Bell wrote:
How do others look at search metrics?
1. Search conversion? Do you look at searches and if the user
does not click on a result, and reruns the search that would be a
failure?
2. How to measure auto complete success metrics?
3. Facets/filters could be considered negative, since we did not
find the results that the user wanted, and now they are filtering
- who to measure?
4. One easy metric is searches with 0 results. We could auto
expand the geo distance or ask the user "did you mean" ?
5. Another easy one would be tech performance: "time it takes in
seconds to get a result".
6. How to measure fuzzy? How do you know you need more synonyms?
How to measure?
7. How many searches it takes before the user clicks on a
result?
Other ideas? Is there a video or presentation on search metrics
that would be useful?
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
