This should have an effect on both ZRR and relevance, but I think the effect will be more on the relevance side than the ZRR side. We could do a manual run through against our labs cluster to check for ZRR changes.
I hadn't thought about it but i suppose if we want to measure relevance that has to be done on the client side to integrate with our satisfaction metrics. The common terms patch supports this use case as well (feature can be enabled via query string) if we want to go that route. On Mon, Oct 5, 2015 at 9:25 AM, Trey Jones <[email protected]> wrote: > David or Erik can correct me if need be, but I think Common Terms is going > to improve results quality more than ZRR. > > Also, we don't have anything that quite lives up to the label of "system" > for re-running queries. We have "methods"—hacks, scripts, and other > non-systematic ways of achieving that goal. [Insert plug for Relevance Lab > here.] > > > Trey Jones > Software Engineer, Discovery > Wikimedia Foundation > > On Mon, Oct 5, 2015 at 12:18 PM, Mikhail Popov <[email protected]> > wrote: > >> 1. Is the common terms query work supposed to address zero results rate >> or…? Don't we have a system now for running old queries through the engine >> and seeing whether the change affected the ZRR? If the test is not >> dependent on the user's actions, we don't need to do an A/B test that >> involves users. >> >> 2. We finally have a couple of ways of assessing user engagement with our >> search results (more on that in an announcement later today). If this work >> is meant to address that, then we need to run an A/B test, but include the >> TestSatisfaction2 schema (or a variant of it that also tracks the grouping). >> >> On Mon, Oct 5, 2015 at 8:59 AM, Erik Bernhardson < >> [email protected]> wrote: >> >>> David's work on the common terms query is merged and will be deploying >>> this week. It's hidden behind a feature flag so no user visible change >>> will occur. >>> >>> I thought we had decided on an A/B test to check the usefulness of this >>> approach but I can't find anything in phab. So the question is, how do we >>> want to AB test this? It will be easiest to run this test through our >>> backend AB testing infrastructure, i think we just need to choose a >>> sampling rate. >>> >>> _______________________________________________ >>> discovery mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/discovery >>> >>> >> >> >> -- >> *Mikhail Popov* // Data Analyst, Discovery >> <https://www.mediawiki.org/wiki/Wikimedia_Discovery> >> https://wikimediafoundation.org/ >> >> *Imagine a world in which every single human being can freely share in >> the **sum of all knowledge. That's our commitment.* Donate >> <https://donate.wikimedia.org/>. >> >> _______________________________________________ >> discovery mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/discovery >> >> > > _______________________________________________ > discovery mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/discovery > >
_______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
