If you want to get deep in to user instructions for Discernatron, check out
Google's:
https://static.googleusercontent.com/media/www.google.com/en//insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf
It's likely a bit excessive, but there are some interesting examples and
their rating scale beginning on page 76.
Rating scale & short descriptions of the scale:
Instructions for how to rate ambiguous queries:
Their judgement UI (or one of):
The map at the top shows where the querying user was located. WMF could do
the same to try to better capture the intent of the user. For example a
user searching for q={national gallery}, is looking for a different wikipage
<https://en.wikipedia.org/wiki/National_Gallery_(disambiguation)> depending
on where they are located.
Feedback button:
Search engine land also has a writeup on Bing's equivalent:
http://searchengineland.com/bing-search-quality-rating-guidelines-130592
--justin
On Thu, Apr 21, 2016 at 4:13 PM, Justin Ormont <[email protected]>
wrote:
There's an example of user instructions for a judgement platform on pages
27-27[sic] of these slides:
http://resources.mpi-inf.mpg.de/d5/teaching/ws14_15/atir/slides/2014-atir-ch09-evaluation.pdf
Judgment instructions generally need a couple of rounds to get right as
they are quite task specific. For example, not defining what to do with a
disambiguation page will cause some judges to rate a page a "1", and some
rate a "3".
--justin
>>
_______________________________________________
discovery mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/discovery
