Hi all, I wrote a telemetry experiment last year[1], and I also found the process challenging to navigate.
I found that many important details were undocumented, but were mentioned in review comments, so I added what I could to the telemetry experiment wiki page and to MDN. My experiment gathered new data, and sent it to a new, one-off server endpoint. This definitely required more discussion than the typical experiment. That said, I do think there are ways the process could be improved for all experiments. Here are a few suggestions for improving the current process: - document how to use Experiments.jsm on MDN - document the schema of the Telemetry Experiment-specific manifest.json file - write and maintain at least one up-to-date, thoroughly commented example experiment - merge (parts of) the telex QA page[2] into the main page (the link is buried in the main page) - update and possibly merge the code docs[3] into the main wiki page To expand on the last bullet point: the code docs suggest using a special python build script to assemble the final .xpi, but that's no longer accurate, as experiments now need to be signed. Further, each experiment has to be *manually* signed, because AMO signing tools will not auto-sign an add-on with the special telex emType. A bug has been filed[4] to enable auto-signing for experiments, but it hasn't moved since November. Might be worth pushing on it. Each of these missing pieces makes shipping an experiment a little bit harder than it has to be. Adding them all up, the process as a whole can seem difficult for a first-time experiment author (at least, it did for me and evidently for Kats as well). I hope these suggestions are helpful. Cheers, Jared [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1174937 [2] https://wiki.mozilla.org/QA/Telemetry [3] http://hg.mozilla.org/webtools/telemetry-experiment-server/file/tip/README.rst [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1220097 On Wed, Apr 20, 2016 at 8:05 AM, Kartikaya Gupta <kgu...@mozilla.com> wrote: > On Wed, Apr 20, 2016 at 10:26 AM, Benjamin Smedberg > <benja...@smedbergs.us> wrote: > > The goal of this is for experiments to be fairly lightweight. > > > > Can you talk about where the problems were? The only signoffs that are > > currently required are code review (no surprise there) and > > acknowledgement from a release driver. > > This sounds reasonable, but the page at > https://wiki.mozilla.org/Telemetry/Experiments (taken at face value, > which is what I did as it was my first foray into this) indicates > otherwise. Perhaps most of my issues could be resolved just by > updating the documentation on that page. For example, it says "Product > approval is required to run an experiment." and is unclear about what > is "user-facing" vs not. It also says to talk to you *before* building > an experiment, which I did (bug 1251052 comment 1), only to then find > out that step was not necessary, so that was extra unnecessary > latency. The doc also says "QA should verify the experience on the > staging server", so I went through that process - it was almost no > work on my part since QA had a template for it already but it still > took nonzero time. The addon signing step is also not yet automated, > as far as I could tell, even though the bug referenced in the doc is > resolved fixed, so that adds an additional dependency and round-trip > to somebody who can sign it. > > > For pref flips in particular, we've talked about extending the > > experiment system so that you don't have to write an addon at all: > > that you can just specify pref values in the experiment manifest. That > > would require engineering work and a little bit of new signing work > > that is currently not planned; but if somebody wants to do that work, > > I would be willing to mentor. > > This sounds great, and would be really nice. If it's not a huge amount > of work I would be willing to take this on. Is there a bug on file for > it? > > > Data review is required only if an experiment collects new data. My > > goal is for this to be fast and straightforward, but IIRC it wasn't > > necessary at all for your recent experiment. There is no legal review > > required for experiments unless I raise a question during data review. > > Again, the wiki page should state this more explicitly, for the > benefit of people who are doing an experiment for the first time. > > > There is no explicit QA "approval" process required: clearly we don't > > want to ship broken code, so we should use normal good judgement about > > how to test each particular experiment, but that should not be a > > high-process thing by default. > > Ditto, wiki page should be clarified. I'm happy to go and update the > page to reflect what you've said here, provided you're willing to > review my changes to make sure I don't go overboard :) > > Cheers, > kats > > > On Tue, Apr 19, 2016 at 4:43 PM, Kartikaya Gupta <kgu...@mozilla.com> > wrote: > >> (Cross-posted to dev-platform and release-management) > >> > >> Hi all, > >> > >> Not too long ago I ran a telemetry experiment [1] to figure out how to > >> tune some of our code to get the best in-the-wild behaviour. While I > >> got the data I wanted, I found the process of getting the experiment > >> going to be very heavyweight as it involved getting all sorts of > >> approvals and reviews. Going through that process was more > >> time-consuming than I would like, and it has put me off from doing > >> further experiments of a similar nature. However, this means that the > >> decisions I make are going to be less data driven and more guesswork, > >> which is not good for obvious reasons. > >> > >> What I would like to see is a simplified process for telemetry > >> experiments on Nightly, making it easier to flip a pref on 50% of the > >> population for a week or two and get some useful data out of it. It > >> seems to me that many of the approvals (QA, RelMan, Legal, Product) > >> should not really be needed for this kind of simple temporary > >> pref-flip, assuming the necessary data collection mechanisms are > >> already in the code. Does anybody have any objections to this, or have > >> other suggestions on how to streamline this process a bit more? > >> > >> To be clear, I'm not suggesting we do away with these approvals > >> entirely, I just want to see more nuance in the process to determine > >> when they are *really* required, so that they don't slow us down > >> otherwise. > >> > >> Cheers, > >> kats > >> > >> [1] https://wiki.mozilla.org/Telemetry/Experiments > >> _______________________________________________ > >> dev-platform mailing list > >> dev-platform@lists.mozilla.org > >> https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
