Re: web-platform-tests now running in automation

2014-09-12 Thread James Graham
On 10/09/14 19:32, Aryeh Gregor wrote: > On Tue, Sep 9, 2014 at 3:44 PM, James Graham wrote: >> Yes, I agree too. One option I had considered was making a suite >> "web-platform-tests-mozilla" for things that we can't push upstream e.g. >> because the APIs aren't (yet) undergoing meaningful standa

Re: web-platform-tests now running in automation

2014-09-10 Thread Aryeh Gregor
On Tue, Sep 9, 2014 at 3:44 PM, James Graham wrote: > Yes, I agree too. One option I had considered was making a suite > "web-platform-tests-mozilla" for things that we can't push upstream e.g. > because the APIs aren't (yet) undergoing meaningful standardisation. > Putting the editing tests into

Re: web-platform-tests now running in automation

2014-09-09 Thread Ehsan Akhgari
On 2014-09-09, 8:44 AM, James Graham wrote: On 08/09/14 19:42, Ehsan Akhgari wrote: I think unreviewed tests should still be run by browsers' automated testing framework (obviously unless they take too long, are unreliable, etc.). They just shouldn't be counted toward any claims of conformance.

Re: web-platform-tests now running in automation

2014-09-09 Thread James Graham
On 08/09/14 19:42, Ehsan Akhgari wrote: >> I think unreviewed tests should still be run by browsers' automated >> testing framework (obviously unless they take too long, are >> unreliable, etc.). They just shouldn't be counted toward any claims >> of conformance. Even if the expected values are e

Re: web-platform-tests now running in automation

2014-09-08 Thread Ehsan Akhgari
On 2014-09-08, 6:47 AM, Aryeh Gregor wrote: On Sun, Sep 7, 2014 at 5:49 PM, James Graham wrote: Well, it would also make sense to have interop for editing of course :) Not a single major browser has significant resources invested in working on their editing code. Until that changes, nothing

Re: web-platform-tests now running in automation

2014-09-08 Thread Aryeh Gregor
On Sun, Sep 7, 2014 at 5:49 PM, James Graham wrote: > Well, it would also make sense to have interop for editing of course :) Not a single major browser has significant resources invested in working on their editing code. Until that changes, nothing much is going to happen. > I would certainly

Re: web-platform-tests now running in automation

2014-09-07 Thread Boris Zbarsky
On 9/7/14, 10:21 AM, James Graham wrote: There isn't anything at the moment, but it seems like a good idea to invent something. The easiest thing would be a new key-value pair like expected-reason: Some reason string Do you have a preferred syntax here? Nope. Pretty much anything works for m

Re: web-platform-tests now running in automation

2014-09-07 Thread James Graham
On 07/09/14 12:34, Aryeh Gregor wrote: > On Fri, Sep 5, 2014 at 8:23 PM, James Graham wrote: >> I think Ms2ger has a better answer here, but I believe it obsoletes most >> of them, except a few that never got submitted to web-platform-tests >> (the editing tests are in that class, because the spec

Re: web-platform-tests now running in automation

2014-09-07 Thread James Graham
On 06/09/14 05:05, Boris Zbarsky wrote: > On 9/5/14, 11:55 AM, James Graham wrote: >> Instructions for performing the updates are in the README file >> [2]. There is tooling available to help in the update process. > > Is there a way to document the spec or test suite bugs in the > expectations fi

Re: web-platform-tests now running in automation

2014-09-07 Thread Aryeh Gregor
On Fri, Sep 5, 2014 at 8:23 PM, James Graham wrote: > I think Ms2ger has a better answer here, but I believe it obsoletes most > of them, except a few that never got submitted to web-platform-tests > (the editing tests are in that class, because the spec effort sort of died). FWIW, the editing te

Re: web-platform-tests now running in automation

2014-09-05 Thread Boris Zbarsky
On 9/5/14, 11:55 AM, James Graham wrote: Instructions for performing the updates are in the README file [2]. There is tooling available to help in the update process. Is there a way to document the spec or test suite bugs in the expectations file? e.g. if I want to add an "expected: FAIL" and

Re: web-platform-tests now running in automation

2014-09-05 Thread James Graham
On 05/09/14 18:00, Boris Zbarsky wrote: > On 9/5/14, 11:55 AM, James Graham wrote: >> The web-platform-tests testsuite has just landed on >> Mozilla-Central. > > This is fantastic. Thank you! > > Does this obsolete our existing "imptests" tests, or is this a set of > tests disjoint from those?

Re: web-platform-tests now running in automation

2014-09-05 Thread Boris Zbarsky
On 9/5/14, 11:55 AM, James Graham wrote: The web-platform-tests testsuite has just landed on Mozilla-Central. This is fantastic. Thank you! Does this obsolete our existing "imptests" tests, or is this a set of tests disjoint from those? -Boris __

Re: web-platform-tests now running in automation

2014-09-05 Thread Kyle Huey
On Fri, Sep 5, 2014 at 8:55 AM, James Graham wrote: > The web-platform-tests testsuite has just landed on > Mozilla-Central. It is an import of a testsuite collated by the W3C > [1], which we intend to keep up-to-date with upstream. The tests are > located in /testing/web-platform/tests/ and are n