Le lun. 15 mai 2017 à 23:15, Maciej Stachowiak <[email protected]> a écrit :
> > On May 15, 2017, at 9:08 PM, youenn fablet <[email protected]> wrote: > > I see two main cases: > - Writer of the patch is making sure to upstream WPT test changes at > WebKit landing time. It is ok to make the changes directly in > LayoutTests/imported/w3c/web-platform-tests/ > - Writer plans to upstream WPT test changes at some point but wants more > time. It is better to develop the tests in LayoutTests/http/wpt and then > migrate them later on. > > > My proposal was different from either of these, it was to have a directory > specifically for tests meant to be upstreamed (LayoutTests/http/wpt should > contain only tests not meant > The possibility to have LayoutTests/http/wpt/to-be-imported was mentionned previously. I think it makes sense to run to-be-imported tests behind WPT as this is the way they will end up being run when upstreamed. > > I think adding new tests directly to > LayoutTests/imported/w3c/web-platform-tests/ is needlessly messy. Most > stuff in the imported/ directory is an exact copy of an upstream test > suite, so if you run only a specific directory of tests, you know you are > running an official conformance suite. With this proposal, it might also > contain random tests that will hopefully be upstreamed but maybe not, or > might be changed before the PR lands upstream, or might get renamed, or > whatever. There's no guarantee that updating from the official version will > ever fully resolve the delta. > LayoutTests/imported/w3c/web-platform-tests is mostly about regression testing. For conformance testing, it is probably more accurate to either use w3c-test.org (probably not reliable enough though) or make a clone of W3C WPT, do the set up as specified in their README (etc/hosts...) and use wptrunner. > > I think it would be more elegant to have a parallel directory > (LayoutTests/for-export/w3c/web-platform-tests). Then when something is > actually upstreamed and then pulled back down, we could delete the staged > version. Directly modifying our local copy seems like it could easily lead > to long-term divergences slipping through the cracks. > A parallel directory is fine when starting a test suite. Often, changes are limited to modifying a test, which might be best done in-place. Often, adding a test is easier by adding it in an already existing file. Tooling should protect us from diverging. Authoring in LayoutTests/for-export/w3c/web-platform-tests might not allow you to use WPT tools and common resources. At upstream time, reviewers will probably tell to use that and that existing resource. There is agreement that a imported/w3c/web-platform-test layout test change that lands in WebKit can be merged upstreamed. I am not sure what would be the process with LayoutTests/for-export/w3c/web-platform-tests tests. > > Of course, people could always go run the official copy from w3c-test.org . > But we usually leave imported conformance test suites unmodified except the > minimum necessary to make them run. > > I would start with an experimental phase with some of us making direct > changes in LayoutTests/imported/w3c/web-platform-tests/. > When we are happy with the tools and think the risk for issues is low > enough (or when the bots can handle most of it for us), hacking > LayoutTests/imported/w3c/web-platform-tests/ could be the default. > > > I think the problem with this plan is not tools risk, it's the fact that a > directory of imported tests can no longer be trusted to actually just be > imported tests. So a smaller number of people doing it to start would not > address my concern. It might be that other people don't care about this. My > opinion counts no more than anyone else's, and I'd be interested in hearing > from others. > I am not sure to understand precisely what the consequences of "imported tests can no longer be trusted to actually just be imported tests" are. Ideally, a WPT test change landing in WebKit should be merged at the same time on W3C WPT. There was a suggestion that LayoutTests/imported/w3c/web-platform-tests be moved to a shorter path like LayoutTests/web-platform-tests. That would also make it clear that this folder is not only about one-way-sync. > > Regards, > Maciej > > > Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa <[email protected]> a écrit : > >> Hi all, >> >> Youenn is working on a patch to automatically create a GitHub PR to >> export tests from a WebKit patch which includes changes to >> web-platform-tests [1]. >> >> That raises a question as to where we should put new tests or modified >> tests intended to be exported to web-platform-tests from WebKit. >> >> I think the most obvious option is to use >> LayoutTests/imported/w3c/web-platform-tests/. However, in the other >> thread about adopting testharness.js (titled Another WPT bite), Maciej >> briefly expressed the preference for creating a new directory: >> https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html >> >> Do other people have strong opinions about this? >> >> - R. Niwa >> >> [1] https://bugs.webkit.org/show_bug.cgi?id=169462 >> _______________________________________________ >> webkit-dev mailing list >> [email protected] >> https://lists.webkit.org/mailman/listinfo/webkit-dev >> > _______________________________________________ > webkit-dev mailing list > [email protected] > https://lists.webkit.org/mailman/listinfo/webkit-dev > >
_______________________________________________ webkit-dev mailing list [email protected] https://lists.webkit.org/mailman/listinfo/webkit-dev
