Not yet, but it's a feature we will probably want to implement. I should file a bug.
Done: https://bugzilla.mozilla.org/show_bug.cgi?id=1201653 On Thu, Sep 3, 2015 at 2:47 PM, Axel Hecht <l...@mozilla.com> wrote: > Can testers self-select which part of the experiment they're seeing? > > I'm thinking about l10n testers in particular, but also in general. > > Axel > > > On 9/3/15 8:16 PM, Mark Finkle wrote: > >> We have decided to start running A/B Testing [1] in Firefox for Android. >> These experiments are intended to optimize specific outcomes, as well as, >> inform our long-term design decisions. We want to create the best Firefox >> experience we can, and these experiments will help. >> >> The system will also allow us to throttle the release of features, called >> staged rollout, so we can monitor new features in a controlled manner >> across a large userbase and a fragmented device ecosystem. If we need to >> rollback a feature for some reason, we'd have the ability to do that. >> >> Technical details: >> * Switchboard is used to control experiment segmenting and staged rollout. >> * Telemetry is used to collect metrics about an experiment. >> * FHR is used to track active experiments so we can correlate to >> application usage. >> >> == What is Switchboard? == >> >> Switchboard is an open source SDK for doing A/B testing and staged >> rollouts >> [2][3]. It connects to a server component, which maintains a list of >> active >> experiments. >> >> The SDK does create a UUID, which is stored on the device. The UUID is >> sent >> to the server, which uses it to "bucket" the client, but the UUID is never >> stored on the server. In fact, the server does not store any data. The >> server we are using is being hosted by Mozilla. >> >> We decided to start using Switchboard because it's simple, open source, >> saves no data and can be hosted by Mozilla. Thanks to the KeepSafe folks >> for releasing Switchboard. >> >> == Planning Experiments == >> >> The Mobile Product and UX teams are the primary drivers for creating >> experiments, but as is common on the Mobile team, ideas can come from >> anywhere. We have been working with the Mozilla Growth team, getting a >> better understanding of how to design the experiments and analyze the >> metrics. UX researchers will also have input into the experiments. >> >> Once Product and UX complete the experiment design, Development would land >> code in Firefox to implement the desired variations of the experiment. >> Development would also land code in the Switchboard server to control the >> configuration of the experiment: Is it active? How are the variations >> distributed across the user population? >> >> Since we use Telemetry to collect metrics on the experiments, the Beta >> channel is likely our best time period to run experiments. Telemetry is on >> by default on Nightly, Aurora and Beta; and Beta is the largest userbase >> of >> those three channels. >> >> Once we decide which variation of the experiment is the "winner", we'll >> change the Switchboard server configuration for the experiment so that >> 100% >> of the userbase will flow through the winning variation. >> >> Yes, a small percentage of the Release channel has Telemetry enabled, but >> it might be too small to be useful for experimentation. Time will tell. >> >> Note: Switchboard itself will be enabled on all channels. It collects no >> data and gives us a "code-free" way of staging rollouts. It much less >> risky >> and time consuming than uplifting patches that need to land on branches at >> specific times. >> >> == What's Happening Now? == >> >> Switchboard has already landed in Nightly [4] and is currently behind a >> Nightly build flag. Once we feel comfortable, we'll let it ride the >> trains. >> Our first experiment will likely be testing a new onboarding experience >> [5]. >> >> [1] https://en.wikipedia.org/wiki/A/B_testing >> [2] https://github.com/KeepSafe/Switchboard >> [3] >> >> http://keepsafe-engineering.tumblr.com/post/28437940369/easy-mobile-ab-testing >> [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1196897 >> [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1199859 >> >> >
_______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev@mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev
