Re: A/B testing with telemetry

2013-12-01 Thread Gavin Sharp
On Thursday, November 28, 2013, Henri Sivonen wrote: > Do we have policy reasons that preclude A/B testing on Release or Beta? > We don't have any policies against it; just some FUD from not having a precedent :) Doing A/B testing on release is something I'm very interested in exploring further.

Re: A/B testing with telemetry

2013-11-28 Thread Henri Sivonen
On Mon, Nov 25, 2013 at 6:59 PM, Lawrence Mandel wrote: >> If not, is there a better way to do this than duplicating probes and >> checking the pref to see which probe should be fed? > > A probe is not restricted to boolean values. You can define a histogram that > maps values to conditions. As s

Re: A/B testing with telemetry

2013-11-25 Thread Taras Glek
Henri Sivonen wrote: Do we have a formalized way to do A/B testing with telemetry? That is, assuming that there is a telemetry probe that measures problem symptoms and a boolean pref for turning on a potential solution, is there a way the declare the pref as something that telemetry queries ca

Re: A/B testing with telemetry

2013-11-25 Thread Lawrence Mandel
- Original Message - > Do we have a formalized way to do A/B testing with telemetry? That is, > assuming that there is a telemetry probe that measures problem > symptoms and a boolean pref for turning on a potential solution, is > there a way the declare the pref as something that telemetr