I added a patch which is in review now [1]. It allows the experiment's segmenting configuration to be used locally, instead of sent from the server.
We only intend to use this when the experiment is in a code path that happens very early in application startup. We lose all ability to dynamically alter the configuration and code path if we use this approach. Any changes must be landed in the client application. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1201384 On Thu, Sep 3, 2015 at 2:25 PM, Martin Thomson <m...@mozilla.com> wrote: > This seems like a good idea. > > Has anyone considered contributing a change to switchboard that would > allow the experiment policy to be downloaded to clients and the A/B > decision made there? That removes any possibility of creating another > potential tracking mechanism. > > On Thu, Sep 3, 2015 at 11:16 AM, Mark Finkle <mfin...@mozilla.com> wrote: > > We have decided to start running A/B Testing [1] in Firefox for Android. > > These experiments are intended to optimize specific outcomes, as well as, > > inform our long-term design decisions. We want to create the best Firefox > > experience we can, and these experiments will help. > > > > The system will also allow us to throttle the release of features, called > > staged rollout, so we can monitor new features in a controlled manner > > across a large userbase and a fragmented device ecosystem. If we need to > > rollback a feature for some reason, we'd have the ability to do that. > > > > Technical details: > > * Switchboard is used to control experiment segmenting and staged > rollout. > > * Telemetry is used to collect metrics about an experiment. > > * FHR is used to track active experiments so we can correlate to > > application usage. > > > > == What is Switchboard? == > > > > Switchboard is an open source SDK for doing A/B testing and staged > rollouts > > [2][3]. It connects to a server component, which maintains a list of > active > > experiments. > > > > The SDK does create a UUID, which is stored on the device. The UUID is > sent > > to the server, which uses it to "bucket" the client, but the UUID is > never > > stored on the server. In fact, the server does not store any data. The > > server we are using is being hosted by Mozilla. > > > > We decided to start using Switchboard because it's simple, open source, > > saves no data and can be hosted by Mozilla. Thanks to the KeepSafe folks > > for releasing Switchboard. > > > > == Planning Experiments == > > > > The Mobile Product and UX teams are the primary drivers for creating > > experiments, but as is common on the Mobile team, ideas can come from > > anywhere. We have been working with the Mozilla Growth team, getting a > > better understanding of how to design the experiments and analyze the > > metrics. UX researchers will also have input into the experiments. > > > > Once Product and UX complete the experiment design, Development would > land > > code in Firefox to implement the desired variations of the experiment. > > Development would also land code in the Switchboard server to control the > > configuration of the experiment: Is it active? How are the variations > > distributed across the user population? > > > > Since we use Telemetry to collect metrics on the experiments, the Beta > > channel is likely our best time period to run experiments. Telemetry is > on > > by default on Nightly, Aurora and Beta; and Beta is the largest userbase > of > > those three channels. > > > > Once we decide which variation of the experiment is the "winner", we'll > > change the Switchboard server configuration for the experiment so that > 100% > > of the userbase will flow through the winning variation. > > > > Yes, a small percentage of the Release channel has Telemetry enabled, but > > it might be too small to be useful for experimentation. Time will tell. > > > > Note: Switchboard itself will be enabled on all channels. It collects no > > data and gives us a "code-free" way of staging rollouts. It much less > risky > > and time consuming than uplifting patches that need to land on branches > at > > specific times. > > > > == What's Happening Now? == > > > > Switchboard has already landed in Nightly [4] and is currently behind a > > Nightly build flag. Once we feel comfortable, we'll let it ride the > trains. > > Our first experiment will likely be testing a new onboarding experience > [5]. > > > > [1] https://en.wikipedia.org/wiki/A/B_testing > > [2] https://github.com/KeepSafe/Switchboard > > [3] > > > http://keepsafe-engineering.tumblr.com/post/28437940369/easy-mobile-ab-testing > > [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1196897 > > [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1199859 > > _______________________________________________ > > dev-planning mailing list > > dev-plann...@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-planning >
_______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev@mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev
