On Monday, March 4, 2013 5:15:56 AM UTC-8, Jim Mathies wrote:
For metrofx we’ve been working on getting omtc and apzc running in the browser.
One of the things we need to be able to do is run performance tests that tell
us whether or not the work we’re doing is having a positive effect on perf. We
currently don’t have automated tests up and running for metrofx and talos is
even farther off.
So to work around this I’ve been putting together some basic perf tests I can
use to measure performance using the mochitest framework. I’m wondering if this
might be a useful answer to our perf tests problems long term.
I think this is an incredibly interesting proposal, and I'd love to see
something like it go forward. Detailed reactions below.
Putting together talos tests is a real pain. You have to write a new test using
the talos framework (which is a separate repo from mc), test the test to be
sure it’s working, file rel eng bugs on getting it integrated into talos test
runs, populated in graph server, and tested via staging to be sure everything
is working right. Overall the overhead here seems way too high.
Yup. And that's a big problem. Not only does this make your life harder, it
makes people not do as much performance testing as they otherwise might. The JS
team has had the experience that adding a new way of creating correctness tests
incredibly easy (with *zero* overhead in the common case) really helped get
more tests written and used. So I think it would be great to make it a lot
easier to write perf tests.
Maybe we should consider changing this system so devs can write performance
tests that suit their needs that are integrated into our main repo? Basically:
1) rework graphs server to be open ended so that it can accept data from test
runs within our normal test frameworks.
IIUC, something like this is a key requirement: letting any perf test feed into
the reporting system. People have pointed out that the Talos tests run on
selected machines, so the perf tests should probably run on them as well,
rather than on the correctness test machines. But that's only a small change to
the basic idea, right?
2) develop of test module that can be included in tests that allows test
writers to post performance data to graph server.
Does that mean a mochitest module? This part seems optional, although certainly
useful. Some tests will require non-mochitest frameworks.
I believe jmaher did some work to get in-browser standard JS benchmarks running
automatically and reporting to graph-server. I'm curious how that would fit in
with this idea--would the test module help at all, or could there be some other
kind of more general module maybe, so that even things like standard benchmarks
can be self-serve?
3) come up with a good way to manage the life cycle of active perf tests so
graph server doesn’t become polluted.
:-) How about getting an owner optionally listed for new tests, and then tests
will be removed if no one is looking at them (according to web server logs) and
there is no owner of record or the owner doesn't say the tests are still needed?
4) port existing talos tests over to the mochitest framework.
5) drop talos.
This seems like it's in the line of "fix Talos". I'm not sure if this
particular 4+5 is the right way to go, but the idea has some merit.
To the extent that people don't pay attention to Talos, it seems we really
don't need to do anything with it. If people are paying attention to and taking
care of performance in their area, then we're covered. To take the example I
happen to know best, the JS team uses AWFY to track JS performance on standard
benchmarks and additional tests they've decided are useful. So Talos is not
needed to track JS performance. Having all the features of the new graph server
does sound pretty cool, though.
It appears that there a few areas that are only covered by Talos for now,
though. I think in that category we have warm startup time via Ts, and basic
layout performance via Tp. I'm not sure about memory, because we do seem to
detect increases via Talos, but we also have AWSY, and I don't know whether
AWSY obviates Talos memory measurements or not.
For that kind of thing, I'm thinking maybe we should go with the same "teams take
care of their own perf tests" idea. Performance is a natural owner for Ts. I'm not
entirely sure about Tp, but it's probably layout or DOM. Then those teams could decide if
they wanted to switch from Talos to a different framework. If everything's working
properly, if the difficulty of reproducing Talos tests locally caused enough problems to
warrant it, the owning teams would notice and switch.
Dave
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
