On 2016-01-11 9:08 AM, jma...@mozilla.com wrote:
Currently we run a very outdated version of V8 (version 7) in Talos. This has
since been replaced with Octane in the world of benchmarks.
AWFY (arewefastyet.com), has been running Octane and catching regressions
faster than Talos. There is missing coverage in AWFY, specifically e10s, pgo,
aurora/beta. There are plans to add coverage for this in Q1.
A main reason for pushing to turn off V8, is that the benchmark is outdated and
regressions could not be the most useful use of developers time if there is a
regression only seen on V8 instead of Octane. While this does point out that
we are leaning towards building performance for a specific benchmark and
ignoring other tests, we could argue that is what we should be doing.
The reason I am posting here is to find out if there are reasons we should keep
v8 running in Talos. We still plan to turn it off once AWFY coverage matches
the coverage of Talos V8.
You can reference bug 1174671 for some history.
It seems like another alternative might be to run Octane in Talos,
instead of v8_7.
It seems like Talos has two advantages over AWFY (correct me if I'm wrong):
1. Easy for developers to schedule jobs via try (maybe less of a concern
with a benchmark like this, where I suspect results are more
reproducible locally?)
2. More hardware available, so can get results faster.
Thoughts? Incidentally one of my deliverables for this quarter is to try
to figure out how Perfherder, Talos, and AWFY should co-exist, so I'm
very interested in knowing if my assumptions above are correct.
Will
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
