Just to close the loop on this, I went ahead and updated the wiki pages at [1] and [2] to reflect that some parts of the process are more optional than they originally seemed. I also tried to generally make the documentation simpler to follow and less redundant/contradictory. Finally, I filed bug 1271440 for the missing piece to allow developers to self-sign their experiment addons.
Hopefully these changes will make it easier to build and deploy experiments in the future. Cheers, kats On Wed, Apr 20, 2016 at 3:31 PM, Jared Hirsch <6...@mozilla.com> wrote: > Hi all, > > I wrote a telemetry experiment last year[1], and I also found the process > challenging to navigate. > > I found that many important details were undocumented, but were mentioned in > review comments, so I added what I could to the telemetry experiment wiki > page and to MDN. > > My experiment gathered new data, and sent it to a new, one-off server > endpoint. This definitely required more discussion than the typical > experiment. That said, I do think there are ways the process could be > improved for all experiments. > > Here are a few suggestions for improving the current process: > - document how to use Experiments.jsm on MDN > - document the schema of the Telemetry Experiment-specific manifest.json > file > - write and maintain at least one up-to-date, thoroughly commented example > experiment > - merge (parts of) the telex QA page[2] into the main page (the link is > buried in the main page) > - update and possibly merge the code docs[3] into the main wiki page > > To expand on the last bullet point: the code docs suggest using a special > python build script to assemble the final .xpi, but that's no longer > accurate, as experiments now need to be signed. Further, each experiment has > to be *manually* signed, because AMO signing tools will not auto-sign an > add-on with the special telex emType. A bug has been filed[4] to enable > auto-signing for experiments, but it hasn't moved since November. Might be > worth pushing on it. > > Each of these missing pieces makes shipping an experiment a little bit > harder than it has to be. Adding them all up, the process as a whole can > seem difficult for a first-time experiment author (at least, it did for me > and evidently for Kats as well). > > I hope these suggestions are helpful. > > Cheers, > > Jared > > > [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1174937 > [2] https://wiki.mozilla.org/QA/Telemetry > [3] > http://hg.mozilla.org/webtools/telemetry-experiment-server/file/tip/README.rst > [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1220097 > > On Wed, Apr 20, 2016 at 8:05 AM, Kartikaya Gupta <kgu...@mozilla.com> wrote: >> >> On Wed, Apr 20, 2016 at 10:26 AM, Benjamin Smedberg >> <benja...@smedbergs.us> wrote: >> > The goal of this is for experiments to be fairly lightweight. >> > >> > Can you talk about where the problems were? The only signoffs that are >> > currently required are code review (no surprise there) and >> > acknowledgement from a release driver. >> >> This sounds reasonable, but the page at >> https://wiki.mozilla.org/Telemetry/Experiments (taken at face value, >> which is what I did as it was my first foray into this) indicates >> otherwise. Perhaps most of my issues could be resolved just by >> updating the documentation on that page. For example, it says "Product >> approval is required to run an experiment." and is unclear about what >> is "user-facing" vs not. It also says to talk to you *before* building >> an experiment, which I did (bug 1251052 comment 1), only to then find >> out that step was not necessary, so that was extra unnecessary >> latency. The doc also says "QA should verify the experience on the >> staging server", so I went through that process - it was almost no >> work on my part since QA had a template for it already but it still >> took nonzero time. The addon signing step is also not yet automated, >> as far as I could tell, even though the bug referenced in the doc is >> resolved fixed, so that adds an additional dependency and round-trip >> to somebody who can sign it. >> >> > For pref flips in particular, we've talked about extending the >> > experiment system so that you don't have to write an addon at all: >> > that you can just specify pref values in the experiment manifest. That >> > would require engineering work and a little bit of new signing work >> > that is currently not planned; but if somebody wants to do that work, >> > I would be willing to mentor. >> >> This sounds great, and would be really nice. If it's not a huge amount >> of work I would be willing to take this on. Is there a bug on file for >> it? >> >> > Data review is required only if an experiment collects new data. My >> > goal is for this to be fast and straightforward, but IIRC it wasn't >> > necessary at all for your recent experiment. There is no legal review >> > required for experiments unless I raise a question during data review. >> >> Again, the wiki page should state this more explicitly, for the >> benefit of people who are doing an experiment for the first time. >> >> > There is no explicit QA "approval" process required: clearly we don't >> > want to ship broken code, so we should use normal good judgement about >> > how to test each particular experiment, but that should not be a >> > high-process thing by default. >> >> Ditto, wiki page should be clarified. I'm happy to go and update the >> page to reflect what you've said here, provided you're willing to >> review my changes to make sure I don't go overboard :) >> >> Cheers, >> kats >> >> > On Tue, Apr 19, 2016 at 4:43 PM, Kartikaya Gupta <kgu...@mozilla.com> >> > wrote: >> >> (Cross-posted to dev-platform and release-management) >> >> >> >> Hi all, >> >> >> >> Not too long ago I ran a telemetry experiment [1] to figure out how to >> >> tune some of our code to get the best in-the-wild behaviour. While I >> >> got the data I wanted, I found the process of getting the experiment >> >> going to be very heavyweight as it involved getting all sorts of >> >> approvals and reviews. Going through that process was more >> >> time-consuming than I would like, and it has put me off from doing >> >> further experiments of a similar nature. However, this means that the >> >> decisions I make are going to be less data driven and more guesswork, >> >> which is not good for obvious reasons. >> >> >> >> What I would like to see is a simplified process for telemetry >> >> experiments on Nightly, making it easier to flip a pref on 50% of the >> >> population for a week or two and get some useful data out of it. It >> >> seems to me that many of the approvals (QA, RelMan, Legal, Product) >> >> should not really be needed for this kind of simple temporary >> >> pref-flip, assuming the necessary data collection mechanisms are >> >> already in the code. Does anybody have any objections to this, or have >> >> other suggestions on how to streamline this process a bit more? >> >> >> >> To be clear, I'm not suggesting we do away with these approvals >> >> entirely, I just want to see more nuance in the process to determine >> >> when they are *really* required, so that they don't slow us down >> >> otherwise. >> >> >> >> Cheers, >> >> kats >> >> >> >> [1] https://wiki.mozilla.org/Telemetry/Experiments >> >> _______________________________________________ >> >> dev-platform mailing list >> >> dev-platform@lists.mozilla.org >> >> https://lists.mozilla.org/listinfo/dev-platform >> _______________________________________________ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform > > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
