As the Firefox data engineering teams we provide core tools for using data to other teams. This spans from collection through *Firefox Telemetry*, storage & processing in our *Data Platform* to making data available in *Data Tools*.
To make new developments more visible we aim to publish a quarterly newsletter. As we skipped one, some important items from Q4 are also highlighted this time. This year our teams are putting their main focus on: - Making experimentation easy & powerful. - Providing a low-latency view into product release health. - Making it easy to work with events end-to-end. - Addressing important user issues with our tools. *Usage improvements* Last year we started to investigate how our various tools are used by people working on Firefox in different roles. From that we started addressing some of the main issues users have. Most centrally, the Telemetry portal <https://telemetry.mozilla.org/> is now the main entry point to our tools, documentation and other resources. When working with Firefox data you will find all the important tools linked from there. We added the probe dictionary <https://telemetry.mozilla.org/probe-dictionary/> to make it easy to find what data we have about Firefox usage. For STMO <https://sql.telemetry.mozilla.org/>, our Redash instance, we deployed a major UI refresh <https://blog.redash.io/improving-redash-user-experience-36e3ca47d1b3> from the upstream project. There is new documentation on prototyping <https://docs.telemetry.mozilla.org/tools/stmo.html#prototyping-queries> and optimizing <https://docs.telemetry.mozilla.org/concepts/sql_optimization.html> STMO queries. Our data documentation <https://docs.telemetry.mozilla.org/> saw many other updates, from cookbooks on how to see your own pings <https://docs.telemetry.mozilla.org/cookbooks/view_pings_cep.html> and sending new pings <https://docs.telemetry.mozilla.org/cookbooks/new_ping.html> to adding more datasets <https://docs.telemetry.mozilla.org/concepts/choosing_a_dataset.html>. We also added documentation on how our data pipeline works <https://docs.telemetry.mozilla.org/concepts/pipeline/data_pipeline.html>. *Enabling experimentation* For experimentation, we have focused on improving tooling. Test Tube <https://firefox-test-tube.herokuapp.com/> will soon be our main experiment dashboard, replacing experiments viewer. It displays the results of multivariant experiments that are being conducted within Firefox. We now have St. Moab <https://github.com/mozilla/stmoab> as a toolkit for automatically generating experiment dashboards. *Working with event data* To make working with events easier, we improved multiple stages in the pipeline. Our documentation has an overview <https://docs.telemetry.mozilla.org/concepts/pipeline/event_pipeline.html> of the data flow. On the Firefox side, events can now be recorded through the events API <https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html>, from add-ons <https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#registerevents>, and whitelisted Firefox content <https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/hybrid-content.html>. From Firefox 61, all recorded events are automatically counted into scalars <https://bugzilla.mozilla.org/show_bug.cgi?id=1440673>, to easily get summary statistics. Event data is available for analysis in Redash in different datasets <https://docs.telemetry.mozilla.org/concepts/pipeline/event_pipeline.html#datasets>. We can now also connect more event data to Amplitude <https://amplitude.com/>, a product analytics tool. A connection for some mobile events to Amplitude is live, for Firefox Desktop events it will be available soon. *Low-latency release health data* To enable low-latency views into release health data, we are working on improving Mission Control <https://data-missioncontrol.dev.mozaws.net/>, which will soon replace arewestableyet.com <https://www.arewestableyet.com/> . It has new features <https://wlach.github.io/blog/2018/04/mission-control-update/> that enable comparing quality measures like crashes release-over-release across channels. *Firefox Telemetry tools* For Firefox instrumentation we expanded on the event recording APIs <https://docs.telemetry.mozilla.org/concepts/pipeline/event_pipeline.html#firefox-event-collection>. To make build turnaround times faster, we now support adding scalars in artifact builds <https://mail.mozilla.org/pipermail/fx-data-dev/2018-February/000114.html> and will soon extend this to events <https://bugzilla.mozilla.org/show_bug.cgi?id=1448945>. Following the recent Firefox data preferences changes, we adopted Telemetry <https://medium.com/georg-fritzsche/data-preference-changes-in-firefox-58-2d5df9c428b5> to only differentiate between "release" and "prerelease" data. This also impacted the measurement dashboard <https://telemetry.mozilla.org/new-pipeline/dist.html> and telemetry.js users as the current approach to publishing this data from the release channel does not work anymore. The measurement dashboard <https://telemetry.mozilla.org/new-pipeline/dist.html> got some smaller usability improvements thanks to a group of contributors. We also prototyped a use counter dashboard <https://georgf.github.io/usecounters/> for easier analysis. *Datasets & analysis tools* To power LetsEncrypt stats <https://letsencrypt.org/stats/>, we publish a public Firefox SSL usage dataset <https://docs.telemetry.mozilla.org/datasets/other/ssl/reference.html>. The following datasets are newly available in Redash or through Spark: - client_counts_daily <https://docs.telemetry.mozilla.org/datasets/batch_view/client_count_daily/reference.html> - This is useful for estimating user counts over a few dimensions and a long history with daily precision. - first_shutdown_summary <https://docs.telemetry.mozilla.org/datasets/batch_view/first_shutdown_summary/reference.html> - A summary of the first main ping of a client’s lifetime. This accounts for clients that do not otherwise appear in main_summary. - churn <https://docs.telemetry.mozilla.org/datasets/mozetl/churn/reference.html> - A pre-aggregated dataset for calculating the 7-day churn for Firefox Desktop. - retention <https://docs.telemetry.mozilla.org/datasets/batch_view/retention/reference.html> - A pre-aggregated dataset for calculating retention for Firefox Desktop. The primary use-case is 1-day retention. For analysis tooling we now have Databricks available <https://groups.google.com/a/mozilla.com/d/msg/databricks-discuss/eEXnfRJaXm0/TgVav-8XBwAJ>. This offers instant-on-notebooks with no more waiting for clusters to spin up and supports Scala, SQL and R. If you're interested sign up to the databricks-discuss mailing list <https://groups.google.com/a/mozilla.com/forum/#!forum/databricks-discuss>. We also got the probe info service <https://github.com/mozilla/probe-scraper> into production, which scrapes the probe data in Firefox code and makes a history of it available to consumers. This is what powers the probe dictionary <https://telemetry.mozilla.org/probe-dictionary/>, but can also be used to power other data tooling. *Getting in touch* Please reach out to us with any questions or concerns. - You can find us on IRC in #telemetry and #datapipeline. - We are available on slack in #fx-metrics. - The main mailing list for data topics is fx-data-dev <https://mail.mozilla.org/listinfo/fx-data-dev>. - Bugs can be filed in one of these components <https://wiki.mozilla.org/Telemetry#Filing_Bugs>. - You can find us on Twitter as @MozTelemetry <https://twitter.com/moztelemetry>. Cheers from - The data engineering team (Katie Parlante), consisting of - The Firefox Telemetry team (Georg Fritzsche) - The Data Platform team (Mark Reid) - The Data Tools team (Rob Miller) _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
