This morning I pushed bug 1252361 to inbound, which modifies the "fuzzy" and "fuzzy-if" reftest annotations to accept ranges in addition to single values for the fuzziness parameters. If a single value "x" is provided, it by default maps to the range "0..x", so it preserves the existing semantics in reftest.list files.
However I would very strongly encourage people adding new reftests with fuzz values to use ranges rather than single values, so that if the actual fuzziness drops we catch it. If the actual pixel differences are below the allowed range, the test will trigger an UNEXPECTED-PASS result, so it should show up on TreeHerder as a failure. It would also be great to look at existing tests and reduce or tighten the allowed fuzziness. There are a lot of tests that are currently "overfuzzed" (i.e. that have larger-than-needed fuzzy values). Here is an example try push [1] that shows tests that are marked as fuzzy but don't need it because they have zero differences (on Linux/Android, anyway). And this is only a subset of the overfuzzed tests. A final thing to note is that the raw logs on TreeHerder will now also print out the exact fuzziness along with the allowed range, so you can easily see from any push what the state of fuzziness is. For example in this log [2] you can find entries like this: REFTEST INFO | REFTEST fuzzy test (0, 0) <= (33, 192) <= (80, 1500) which indicates that the *actual* differences in the test/ref were (33,192) but the allowed range was (0..80, 0..1500). So in this case the test is overfuzzed because it allows a difference of up to 1500 pixels but only 192 pixels are different (and likewise it allows any rgb component to be different by up to 80, but the maximum encountered difference was 33). This was a Linux64 log, and we'd want to check the results for this test across all platforms, but it's possible this range could be tightened. See the commits on the bug for more documentation on the exact semantics of the ranges. Cheers, kats [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=63d6def39a86e54db467a79fd9be71b5ca9be41b [2] https://treeherder.mozilla.org/logviewer.html#?job_id=104071750&repo=mozilla-inbound _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
