On 09/01/16 22:29, James Graham wrote:
At this point I don't see any real advantages to trying to move to
manifestparser for all web-platform-tests and many drawbacks, so I don't
think it will happen. I am also not convinced that it's very relevant to
the problem at hand; I don't see how the different manifest format is
causing any issues. Indeed, now that most of our testsuites produce
structured log output, you don't actually need to look at the input
manifests at all.
The right thing to do is look at the log files produced from a test run.
This is what wptview provides a GUI for, and what the test-informant
tool ahal mentioned elsewhere does on the backend, but anyone with a
little bit of time and a basic knowledge of the mozlog format (and
treeherder API, perhaps) could have a go at making a one-off tool to
answer this specific question efficiently. To do this one would consume
all the structured logs for the e10s and non-e10s jobs on a push, and
look for cases where the result is different for the same test in the
two run types (this would also cover disabled tests that are recorded as
SKIP).
So I wrote a script to do this i.e. to produce a complete list of the
differences between e10s and non-e10s results in a given treeherder run.
You can get it at [1].
The repository also contains some sample output from a randomly selected
treeherder run; the .txt file is designed for human consumption; each
line has the format
test name | subtest name | non-e10s result | e10s result | platforms
Note that reftest harness based test types are currently not supported
pending bug 1034290.
[1] https://github.com/jgraham/e10s-compare
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
