On 08/01/16 22:41, Robert O'Callahan wrote:
On Sat, Jan 9, 2016 at 10:27 AM, Benjamin Smedberg <benja...@smedbergs.us>
wrote:
What are the implications of this?
The web-platform tests are pass/fail, right? So is it a bug if they pass
but have different behaviors in e10s and non-e10s mode?
Yeah, I'm confused.
If a wpt test passes but with different output, then either there is no
problem or the test is incomplete and should be changed.
Maybe I should clarify.
web-platform-tests are slightly different to most tests in that we run
both tests we currently pass and tests that we currently don't pass. On
treeherder all we check is that we got the same result in this run as we
expected on the basis of previous runs. That result might be pass but
might also be FAIL, ERROR, TIMEOUT, or even CRASH. So they are pass/fail
from the point of view of "did we meet the expectation value", but the
expectation value itself might not be a PASS (e.g. expected FAIL got
PASS would turn treeherder orange, as would expected CRASH got ERROR).
For e10s runs we have the ability to set different expectation values
than for non-e10s runs. This means that we can continue to run tests
that behave differently in e10s an only disable unstable ones. This has
the advantage that we will catch some additional types of regression
e.g. one that causes a test that PASSes in non e10s, previously FAILed
in e10s and starts to CRASH in e10s whilst still PASSing in non-e10s.
These would be missed if we just disabled all tests will differing
behaviour.
The effect of all of this is that in order to understand what's actually
needed to bring e10s builds up to par with non-e10s builds you need to
look at the actual test results rather than just the list of disabled
tests. I believe that there are both instances of tests that pass in
non-e10s but not in e10s builds, and the reverse. wptview gives you the
ability to do that using data directly from treeherder. The actual
action to take on the basis of this data is obviously something for the
people working on e10s to determine.
I hope that clarifies things somewhat?
Whilst I am here, it's always worth calling out contributions; wptview
is a Kalpesh's ateam "Quarter of Contribution" project and he has done
great work.
P.S. I am currently on leave and will remain so until 18th Jan, so don't
be surprised if I am unresponsive to follow-ups until then. Ms2ger is a
good person to ask web-platform-tests questions to in the interim.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
