The goal of this is for experiments to be fairly lightweight. Can you talk about where the problems were? The only signoffs that are currently required are code review (no surprise there) and acknowledgement from a release driver.
For pref flips in particular, we've talked about extending the experiment system so that you don't have to write an addon at all: that you can just specify pref values in the experiment manifest. That would require engineering work and a little bit of new signing work that is currently not planned; but if somebody wants to do that work, I would be willing to mentor. Data review is required only if an experiment collects new data. My goal is for this to be fast and straightforward, but IIRC it wasn't necessary at all for your recent experiment. There is no legal review required for experiments unless I raise a question during data review. There is no explicit QA "approval" process required: clearly we don't want to ship broken code, so we should use normal good judgement about how to test each particular experiment, but that should not be a high-process thing by default. In what ways does your experience differ from the ideal? What can we change to make this less painful? --BDS On Tue, Apr 19, 2016 at 4:43 PM, Kartikaya Gupta <kgu...@mozilla.com> wrote: > (Cross-posted to dev-platform and release-management) > > Hi all, > > Not too long ago I ran a telemetry experiment [1] to figure out how to > tune some of our code to get the best in-the-wild behaviour. While I > got the data I wanted, I found the process of getting the experiment > going to be very heavyweight as it involved getting all sorts of > approvals and reviews. Going through that process was more > time-consuming than I would like, and it has put me off from doing > further experiments of a similar nature. However, this means that the > decisions I make are going to be less data driven and more guesswork, > which is not good for obvious reasons. > > What I would like to see is a simplified process for telemetry > experiments on Nightly, making it easier to flip a pref on 50% of the > population for a week or two and get some useful data out of it. It > seems to me that many of the approvals (QA, RelMan, Legal, Product) > should not really be needed for this kind of simple temporary > pref-flip, assuming the necessary data collection mechanisms are > already in the code. Does anybody have any objections to this, or have > other suggestions on how to streamline this process a bit more? > > To be clear, I'm not suggesting we do away with these approvals > entirely, I just want to see more nuance in the process to determine > when they are *really* required, so that they don't slow us down > otherwise. > > Cheers, > kats > > [1] https://wiki.mozilla.org/Telemetry/Experiments > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
