On Thursday, August 30, 2012 9:05:40 PM UTC-4, Ben Hearsum wrote: > On 08/30/12 07:22 PM, L. David Baron wrote: > > > On Thursday 2012-08-30 14:42 -0700, Taras Glek wrote: > > >> * Joel will revisit maintaining Talos within mozilla-central to > > >> reduce developer barriers to understanding what a particular Talos > > >> test result means. This should also make Talos easier to run > > > > > > This will also solve one of the other problems that leads developers > > > to distrust talos, which is that a significant portion of the > > > performance regressions reported are (or at least were at one time) > > > the result of changes to the tests, but that changes to the tests > > > don't show up as part of the list of suspected causes of > > > regressions. > > > > This isn't true anymore, actually. While Talos itself isn't stored in > > mozilla-central, a pointer to a specific version of it is. The test > > machines pull the Talos version specified in > > https://mxr.mozilla.org/mozilla-central/source/testing/talos/talos.json > > at test time. For example (from > > https://tbpl.mozilla.org/php/getParsedLog.php?id=14852731&tree=Firefox&full=1): > > /tools/buildbot/bin/python talos_from_code.py --talos-json-url > > http://hg.mozilla.org/mozilla-central/raw-file/f972f1a71e7e/testing/talos/talos.json > > > > This means that changes to the Talos suite *are* associated with a > > mozilla-central revision, have tests run for them, can be backed out, > > can ride trains, etc.
I have backed out changes made to talos and the tests a few times due to performance regressions. While I might not catch every one, we do treat talos changes as another changeset in m-c. If there is an expected shift in numbers, we create a new test. This is why there are 5+ versions of all the tests. It really adds a lot of overhead and breakage (e.g. compare-talos), but this way we don't confuse the old test data with the new adjusted tests. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
