On 12-08-16 11:34 AM, Ms2ger wrote:
On 08/16/2012 05:10 PM, Ehsan Akhgari wrote:
I think this is generally a good idea. I have a few questions before
jumping in to agree though.
1. Is the current testharness.js API the documentation at the beginning
of <http://w3c-test.org/resources/testharness.js>? If that is the case,
the API looks a lot heavier weight than the default mochitest API we
use. In that case, would it make sense for us to have a compatibility
layer which translates our mochitest APIs to the equivalent
testharness.js API calls? (I'm not 100% sure if that conversion would
be straightforward.)
I don't feel it's terribly heavyweight. In the simple case, you need a
test(function() {
// tests
});
That's two extra lines of boilerplate. Our mochitest boilerplate is…
(*looks it up*) 30 lines. I think we'll deal. The method names for
assertions (assert_*) are a bit longer than we're used to, but in my
experience, you get used to it rather quickly.
I was not necessarily talking about the size of the boilerblate. I was
mostly talking about the size of the API. The mochitest API is
considerably smaller than testharness.js, and is therefore easier to
read and learn, I think.
IMO, it would not be a good idea to try to layer another API on top of
this, because it makes our tests harder to understand and less reusable
by other browser vendors, and it means that experience with our brand of
testharness.js doesn't help much to understand the standard brand.
I would agree if the mochitest API was also huge. We're basically
talking about 3 functions: is, is_not and ok. I don't agree that using
these functions would make our tests any less usable by other browser
vendors. But I understand that in their eyes, these functions may not
be as intuitive as in mine. :-)
I also don't think it would be terribly straightforward, and I think
that having real mochitests, testharness.js-that-look-like-mochitest and
standard testharness.js in our tree would make our testing situation
more confusing than the lessened learning curve would be worth. In
particular, having two APIs that superficially look the same but are
actually built on unrelated frameworks seems a recipe for a lot of
annoying differences in edge cases.
Hrm, which edge cases are you talking about? Let's talk about is(a, b,
"msg") for example. Why do you think there would be cases where calling
is() is less understandable or more error prone than assert_equals?
On the testharness.js side, we have things like assert_regexp_match, for
example. I would argue that whether or not assert_regexp_match(a,
/foo/, "msg") is more readable than ok(/foo/.match(a), "msg") is very
subjective and depends on what the author of the test is used to see.
2. Is there any support for running reftest-style tests in a framework
that is reusable by other browsers? If not, can we move to propose the
reftest framework to the appropriate standards bodies so that it can be
adopted by other browsers? Our reftest framework has been carefully
designed to be Gecko-agnostic, and is much superior to the equivalent
testing framework that WebKit has (not sure about other browser
engines). Furthermore, the files loaded by this framework are not
loaded in a privileged context with APIs such as SpecialPowers, which
makes a large number of them portable to other browser engines.
The W3C has already adopted reftests. See
<https://test.csswg.org/harness/test/CSS-STYLE-ATTR_DEV/single/style-attr-braces-001/>
for example.
In fact, we've already got a place in m-c to put reftests to be
submitted to the CSSWG:
<http://mxr.mozilla.org/mozilla-central/source/layout/reftests/w3c-css/submitted/README?force=1>.
Oh, that's good news! Thanks for the pointer.
The main issue here is that the CSSWG appears to impose rather stringent
test formatting requirements, which makes writing reftests for them much
more of a drag than just writing them for ourselves.
I'm assuming that you're talking about
<http://wiki.csswg.org/test/format>. Yeah, that's quite verbose and
cumbersome. :(
I think it makes sense for us if we can start this effort on the reftest
framework, since that has a much lower barrier to entry, and ultimately
this effort would be valuable only if other browser engines start to use
our tests (and hopefully share theirs with us as well).
Opera already shares a lot of their tests, and Google, Apple and
Microsoft have been known to submit tests as well, some of which are
already running on tinderbox.
Great!
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
