(CCing auto-to...@mozilla.com)
jmaher and jhammel will be able to comment more on the talos specifics,
but few thoughts off the top of my head:
It seems like we're conflating multiple issues here:
1) "[talos] is a separate repo from mc"
2) "[it's a hassle to] test the test to be sure it’s working"
3) "[it's a hassle to get results] populated in graph server"
4) "[we need to] come up with a good way to manage the life cycle of
active perf tests so graph server doesn’t become polluted"
Switching from the talos harness to mochitest doesn't fix #2 (we still
have to test, and I don't see how it magically becomes any easier
without extra work - that could have been applied to talos instead) or
#3/#4 (orthogonal problem). It also seems like a brute force way of
fixing #1 (we could just check talos into mozilla-central).
Instead, I think we should be asking:
1) Is the best test framework for performance testing: [a] talos (with
improvements), [b] mochitest (with a significant amount of work to make
it compatible), or [c] a brand new framework?
2) Regardless of framework used, would checking it into mozilla-central
improve dev workflow enough to outweigh the downsides (see bug 787200
for history on that discussion)?
3) Regardless of framework used, how can we make the
development/testing/staging cycle less painful?
4) Regardless of framework used, who should be responsible for ensuring
we actively prune performance tests that are no longer relevant?
Note also that graphs.mozilla.org will be depreciated soon, in favour
of datazilla - which afaik is less painful for adding new test suites
(eg doesn't need manual database changes); jeads can say more on that
front.
Best wishes,
Ed
On 04 March 2013 13:15:56, Jim Mathies wrote:
For metrofx we’ve been working on getting omtc and apzc running in the browser.
One of the things we need to be able to do is run performance tests that tell
us whether or not the work we’re doing is having a positive effect on perf. We
currently don’t have automated tests up and running for metrofx and talos is
even farther off.
So to work around this I’ve been putting together some basic perf tests I can
use to measure performance using the mochitest framework. I’m wondering if this
might be a useful answer to our perf tests problems long term.
Putting together talos tests is a real pain. You have to write a new test using
the talos framework (which is a separate repo from mc), test the test to be
sure it’s working, file rel eng bugs on getting it integrated into talos test
runs, populated in graph server, and tested via staging to be sure everything
is working right. Overall the overhead here seems way too high.
Maybe we should consider changing this system so devs can write performance
tests that suit their needs that are integrated into our main repo? Basically:
1) rework graphs server to be open ended so that it can accept data from test
runs within our normal test frameworks.
2) develop of test module that can be included in tests that allows test
writers to post performance data to graph server.
3) come up with a good way to manage the life cycle of active perf tests so
graph server doesn’t become polluted.
4) port existing talos tests over to the mochitest framework.
5) drop talos.
Curious what people think of this idea.
Jim
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
