On 08/17/2012 01:22 PM, Aryeh Gregor wrote:
If there's one big problem with shared tests, it's that we have to
change the way we annotate expected failures. Currently we just go in
and change ok() to todo() or whatever in the source code of the test,
but of course that doesn't work for shared tests. testharness.js
expects you to break things up into test()s of one or more
assert_*()s, with the same number of test()s running no matter what,
and each test() either passes or fails as a unit. Then you have to
keep track of expected failures out-of-band (see files in
dom/imptests/failures/). The major disadvantage of this is that if a
test() tests multiple things and one of them is expected to fail, you
lose regression-testing for any subsequent ones, because the test()
aborts at the first assert failure.
So in practice, it's not always clear where to divide up your test()s.
One assert per test would be best for regression-testing, but it adds
a lot of clutter to the source code. I think this is the one big
thing that makes testharness.js more complicated to use than
mochitest, although it's still not rocket science. If we decide
test-per-assert is the way to go, perhaps we could get a series of
functions added to make single-assert tests simpler.
James, could you explain again what exactly the benefit is of this
test/assert distinction? Mochitests effectively have one assert per
test hardcoded in, and they work fine for us.
So the theory is that a test can have preconditions that are needed for
the rest of the test to make sense. For example if you are writing a
history navigation test you might depend on a particular action adding
an entry to the session history to get to the state where you can start
your real test. If that fails you want the test to abort before it
starts trying to manipulate the erroneous state. Hence the design that
allows multiple asserts per logical test.
I think for the way that Mozilla track regressions it works OK to just
abort the whole file at the first sign of failure. That probably makes
the Mochitest approach acceptable. however it doesn't work well for
everyone. In particular it doesn't meet Opera's needs very well.
I agree that reftests would be easier to share, though. Crashtests
would be even easier! But mochitests are really where most of our
tests are. Also, unlike reftests, they can mostly be run in the
browser with no special privileges. But as far as the actual
sharing-tests thing goes, yes, it would make sense to start any kind
of sharing initiative with crashtests, then reftests, then mochitests.
FWIW we already use some of your reftests. I think our import is rather
out of date though, possibly dating from when we first implemented
reftests and needed some to test the system. Our implementation is not
100% compatible with yours; for example we don't use reftest-wait but
have special magic in our harness that achieves something similar in an
entirely different way, but essentially the interoperability is very
good. So if you put reftests somewhere it is easy for us to import them
we will happily make use of them.
On Thu, Aug 16, 2012 at 8:20 PM, L. David Baron <dba...@dbaron.org> wrote:
It's two extra lines of boilerplate if you only have one test in the
file.
But if you have many tests in the file, it's a lot more, since each
test needs to be wrapped in this -- at least in my understanding.
Some browser vendors (e.g., Opera) seem to care quite strongly that
each test file always execute the same number of tests in the same
order -- even if some of those tests fail by throwing an exception.
(We don't actually care about order. But we do care that the same test
file always runs the same tests).
The W3C already has test suites we can submit to in testharness.js
format. We run some of those tests as mochitests; I know Opera does
as well. I believe WebKit doesn't run them automatically yet. James
Graham of Opera has indicated that they'd probably be interested in
running our tests. (Opera gets much less user testing than we do, so
they're very interested in automated testing.)
Yes, we are interested in running your tests and very much hope that you
are interested in running the tests that we write (we are trying, with
some success, to share our testsuites with the W3C, although the process
is not yet as slick as I would like).
The relative amount of user testing various implementations get isn't
really relevant; we like automated testing because it allows us to find
bugs seconds to hours after the code is written rather than days or
weeks. We like sharing tests because it improves the web platform as a
whole by encouraging browsers to converge on interoperable behaviours.
There are currently wild differences between UAs in some areas, and that
doesn't benefit anyone.
The basic idea here sounds worthy, but one concern is that our own tests are
often unreliable in our own browser -- and I'd expect that to only get worse
as other browsers and their tests enter the picture. I'd therefore suggest
that a successful cross-browser test effort should prioritize getting stuff
running (even with just a handful of tests)... That way fun problems like
reliability have a chance to be found/fixed over time, instead of having a
megatestsuite suddenly appear that's unappealing to get working.
Yes, I think it would be a good idea to start small.
FWIW we try to automatically detect unreliable tests and disable them
until they are fixed. This makes it possible to deal with large test
dumps without many problems. I don't know quite how that would translate
to your infrastructure.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
