On Wed, Apr 20, 2016 at 10:26 AM, Benjamin Smedberg <benja...@smedbergs.us> wrote: > The goal of this is for experiments to be fairly lightweight. > > Can you talk about where the problems were? The only signoffs that are > currently required are code review (no surprise there) and > acknowledgement from a release driver.
This sounds reasonable, but the page at https://wiki.mozilla.org/Telemetry/Experiments (taken at face value, which is what I did as it was my first foray into this) indicates otherwise. Perhaps most of my issues could be resolved just by updating the documentation on that page. For example, it says "Product approval is required to run an experiment." and is unclear about what is "user-facing" vs not. It also says to talk to you *before* building an experiment, which I did (bug 1251052 comment 1), only to then find out that step was not necessary, so that was extra unnecessary latency. The doc also says "QA should verify the experience on the staging server", so I went through that process - it was almost no work on my part since QA had a template for it already but it still took nonzero time. The addon signing step is also not yet automated, as far as I could tell, even though the bug referenced in the doc is resolved fixed, so that adds an additional dependency and round-trip to somebody who can sign it. > For pref flips in particular, we've talked about extending the > experiment system so that you don't have to write an addon at all: > that you can just specify pref values in the experiment manifest. That > would require engineering work and a little bit of new signing work > that is currently not planned; but if somebody wants to do that work, > I would be willing to mentor. This sounds great, and would be really nice. If it's not a huge amount of work I would be willing to take this on. Is there a bug on file for it? > Data review is required only if an experiment collects new data. My > goal is for this to be fast and straightforward, but IIRC it wasn't > necessary at all for your recent experiment. There is no legal review > required for experiments unless I raise a question during data review. Again, the wiki page should state this more explicitly, for the benefit of people who are doing an experiment for the first time. > There is no explicit QA "approval" process required: clearly we don't > want to ship broken code, so we should use normal good judgement about > how to test each particular experiment, but that should not be a > high-process thing by default. Ditto, wiki page should be clarified. I'm happy to go and update the page to reflect what you've said here, provided you're willing to review my changes to make sure I don't go overboard :) Cheers, kats > On Tue, Apr 19, 2016 at 4:43 PM, Kartikaya Gupta <kgu...@mozilla.com> wrote: >> (Cross-posted to dev-platform and release-management) >> >> Hi all, >> >> Not too long ago I ran a telemetry experiment [1] to figure out how to >> tune some of our code to get the best in-the-wild behaviour. While I >> got the data I wanted, I found the process of getting the experiment >> going to be very heavyweight as it involved getting all sorts of >> approvals and reviews. Going through that process was more >> time-consuming than I would like, and it has put me off from doing >> further experiments of a similar nature. However, this means that the >> decisions I make are going to be less data driven and more guesswork, >> which is not good for obvious reasons. >> >> What I would like to see is a simplified process for telemetry >> experiments on Nightly, making it easier to flip a pref on 50% of the >> population for a week or two and get some useful data out of it. It >> seems to me that many of the approvals (QA, RelMan, Legal, Product) >> should not really be needed for this kind of simple temporary >> pref-flip, assuming the necessary data collection mechanisms are >> already in the code. Does anybody have any objections to this, or have >> other suggestions on how to streamline this process a bit more? >> >> To be clear, I'm not suggesting we do away with these approvals >> entirely, I just want to see more nuance in the process to determine >> when they are *really* required, so that they don't slow us down >> otherwise. >> >> Cheers, >> kats >> >> [1] https://wiki.mozilla.org/Telemetry/Experiments >> _______________________________________________ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
