I've been putting the up-to-date CSSWG tests through their paces in
https://github.com/servo/servo/pull/19070/ and I have good news to
report. We do not appear to be losing test coverage due to the pervasive
use of XHTML; I separated the commit that copied over our existing
metadata and the commit that includes the updated expected results so
it's obvious how our existing results were affected. Our coverage
actually improved in some respects, because the newer copies of the
tests fix a bunch of _really silly_ test mistakes that mean we pass more
tests now and can catch more regressions as a result.
Most common reasons for new test failures in
https://github.com/servo/servo/pull/19070/commits/1d669fc6f46861d554254d41b3c3d3c0a9552acf
include:
* tests that did not used to run in test-css (such as XHTML-only tests)
* directories that did not used to run that are now part of a merged
directory that runs (such as css-color4_dev)
* new tests that did not exist in our old copy
* tests that are no longer categorized as manual tests in the newer harness
I've summarized the effects this new testsuite will have on our build
times in
https://github.com/servo/servo/pull/19070#issuecomment-341127618 . I
propose that we make the builders that currently run the test-css jobs
take over part of test-wpt, which should keep the mac build times down.
We should be able to remove the linux-rel-css builder entirely and
increase our capacity a bit, too.
In short, I think we should enable the new testsuite, remove the old
tests and related CI jobs, and never look back! The ability to upstream
reftests and share others' improvements will be a significant boon.
Cheers,
Josh
_______________________________________________
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo
