This seems like a good idea. Has anyone considered contributing a change to switchboard that would allow the experiment policy to be downloaded to clients and the A/B decision made there? That removes any possibility of creating another potential tracking mechanism.
On Thu, Sep 3, 2015 at 11:16 AM, Mark Finkle <mfin...@mozilla.com> wrote: > We have decided to start running A/B Testing [1] in Firefox for Android. > These experiments are intended to optimize specific outcomes, as well as, > inform our long-term design decisions. We want to create the best Firefox > experience we can, and these experiments will help. > > The system will also allow us to throttle the release of features, called > staged rollout, so we can monitor new features in a controlled manner > across a large userbase and a fragmented device ecosystem. If we need to > rollback a feature for some reason, we'd have the ability to do that. > > Technical details: > * Switchboard is used to control experiment segmenting and staged rollout. > * Telemetry is used to collect metrics about an experiment. > * FHR is used to track active experiments so we can correlate to > application usage. > > == What is Switchboard? == > > Switchboard is an open source SDK for doing A/B testing and staged rollouts > [2][3]. It connects to a server component, which maintains a list of active > experiments. > > The SDK does create a UUID, which is stored on the device. The UUID is sent > to the server, which uses it to "bucket" the client, but the UUID is never > stored on the server. In fact, the server does not store any data. The > server we are using is being hosted by Mozilla. > > We decided to start using Switchboard because it's simple, open source, > saves no data and can be hosted by Mozilla. Thanks to the KeepSafe folks > for releasing Switchboard. > > == Planning Experiments == > > The Mobile Product and UX teams are the primary drivers for creating > experiments, but as is common on the Mobile team, ideas can come from > anywhere. We have been working with the Mozilla Growth team, getting a > better understanding of how to design the experiments and analyze the > metrics. UX researchers will also have input into the experiments. > > Once Product and UX complete the experiment design, Development would land > code in Firefox to implement the desired variations of the experiment. > Development would also land code in the Switchboard server to control the > configuration of the experiment: Is it active? How are the variations > distributed across the user population? > > Since we use Telemetry to collect metrics on the experiments, the Beta > channel is likely our best time period to run experiments. Telemetry is on > by default on Nightly, Aurora and Beta; and Beta is the largest userbase of > those three channels. > > Once we decide which variation of the experiment is the "winner", we'll > change the Switchboard server configuration for the experiment so that 100% > of the userbase will flow through the winning variation. > > Yes, a small percentage of the Release channel has Telemetry enabled, but > it might be too small to be useful for experimentation. Time will tell. > > Note: Switchboard itself will be enabled on all channels. It collects no > data and gives us a "code-free" way of staging rollouts. It much less risky > and time consuming than uplifting patches that need to land on branches at > specific times. > > == What's Happening Now? == > > Switchboard has already landed in Nightly [4] and is currently behind a > Nightly build flag. Once we feel comfortable, we'll let it ride the trains. > Our first experiment will likely be testing a new onboarding experience [5]. > > [1] https://en.wikipedia.org/wiki/A/B_testing > [2] https://github.com/KeepSafe/Switchboard > [3] > http://keepsafe-engineering.tumblr.com/post/28437940369/easy-mobile-ab-testing > [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1196897 > [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1199859 > _______________________________________________ > dev-planning mailing list > dev-plann...@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-planning _______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev@mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev
