These are all really great. Thanks, James! On 12/04/2019 12:11, James Graham wrote: > There have been a few recent changes related to wpt that may be of > interest to a wider audience; for brevity I'm coalescing them into a > single email: > > * New wpt dashboard focused on interop problems > * Support for fuzzy annotations in reftests > * Better support for debugging failing reftests on wpt.fyi > > == Interop Dashboard == > > https://jgraham.github.io/wptdash/ (url may change) is a dashboard > intended to allow identifying web-platorm-test failures that present an > web-compat risk. > > By default it shows tests that are failing in Gecko but passing in Both > Blink and WebKit, based on wpt.fyi runs of Firefox, Chrome and Safari. > Tests are divided by bug component. > > In addition it presents a summary of the in-tree metadata in expectation > ini files to provide an overview of where tests are disabled in gecko, > and various test problems that aren't visible in wpt.fyi data (e.g. > leaks, or debug-only crashes). > > There is upstream worked planned to add a system for annotating test > results from wpt.fyi; this will allow us to associate each gecko-only > failure with a bug and eventually to ensure that we triage all failures > that look like interop issues. > > Please let me know about any changes that would make the dashboard more > useful to you. > > == Fuzzy annotations available in wpt reftests == > > wpt has adopted a system to mark reftests as fuzzy. This is semantically > identical to the system used by gecko reftests; the fuzziness annotation > consists of two ranges; one for the number of pixels that may contain > differences and one for the maximum difference in any colour channel. > > In the case where the test is known to (possibly) be an inexact match in > any configuration, the annotation may be put directly in the test file > as a <meta name=fuzzy> element e.g. > > <meta name=fuzzy content="maxDifference=10-15;totalPixels=300"> > > (note that as with the recent changes to reftests, this requires exactly > 300 pixels different i.e. the implied range is 300-300 not 0-300). > > More usually, however the fuzziness will be browser and configuration > specific; in this case the annotation must be put in the wpt expectation > ini file (i.e. the one under testing/web-platform/meta). In this case > the basic syntax is: > > [test.html] > fuzzy: maxDifference=8;totalPixels=0-1 > > The "if" syntax used for configuration-specific expectations also works > in this case. > > In reftests involving multiple possible references or long chains of > references it may be necessary to specify exactly which comparison > requires the annotation or have multiple comparisons with different > annotations. This can be done as in the example below: > > [test.html] > fuzzy: [ref1.html:maxDifference=8;totalPixels=0-1, > ref2.html==ref3.html:maxDifference=4;totalPixels=0-2] > > The first annotation in the list applies to any comparison involving > ref1.html as the reference, the second only applies to the specific > comparison ref2.html == ref3.html (all paths are resolved relative to > the test). > > More documentation is available at [1] and [2], the latter should > obviously move somewhere more useful. > > == Reftest Comparisons == > > Debugging wpt reftest failures should now be easier since, thanks to > Chrome's ecosystem-infra team, wpt.fyi has gained reftest-analyzer like > functionality. See [3] for an example. wpt.fyi shows links to the > analyzer for all failing reftests [4]. Again, feedback about > improvements to make this more useful is very much encouraged. > > > [1] > https://web-platform-tests.org/writing-tests/reftests.html#fuzzy-matching > [2] > https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/manifestexpected.py#102 > [3] > https://wpt.fyi/analyzer?screenshot=sha1%3A78ed15d1532e4134d8e3560c060538fd0b0a80d9&screenshot=sha1%3A5b8f16d25cb907619183b551c6b3e4d670991268 > [4] > https://wpt.fyi/results/css/css-backgrounds/background-image-first-line.html?label=master&label=experimental&product=chrome&product=firefox&product=safari >
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
