I've also seen TV failures caused by flaws in the test harness which weren't visible before. See bug 1410165 [1] for example, it was caused by an observer being unregistered after the first iteration of a test had finished, causing the following iterations to fail.
Gabriele
[1] Permaorange test-verify
dom/plugins/test/mochitest/test_crash_notify.xul |
expected-crash-dump-missing - This test did not leave any crash dumps
behind, but we were expecting some!
https://bugzilla.mozilla.org/show_bug.cgi?id=1410165
On 06/03/18 20:05, Kyle Machulis wrote:
> In my experience, many tests were written pre-test-verify, and don't clean
> up correctly to deal with multiple runs in the same process. They work fine
> when running as a single session, but blow up in TV. Having skip-if(verify)
> means that we can at least mark those tests as known broken on TV without
> taking them out of the test suites completely, especially if we find them
> in places that we don't know how to fix them and need to file to someone
> else who may not currently have time to fix.
>
> On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez <emi...@crisal.io>
> wrote:
>
>> On 03/06/2018 06:04 PM, Geoffrey Brown wrote:
>>> It is now possible to skip tests in test-verify. Simplify annotate the
>>> manifest for your test:
>>>
>>> [test]
>>> skip-if = verify
>>>
>>> or, for reftests:
>>>
>>> skip-if(verify) ...
>>>
>>> and the test-verify (TV) test task will not try to verify the annotated
>>> test.
>>>
>>> Please don't abuse this feature! Most TV failures indicate a weakness in
>>> the test.
>>
>> Could you point to an example of a "good" use of this feature? Is it
>> just to avoid TV failing too intermittently? Is TV unable to run some
>> tests? When does TV provide no value?
>>
>> Thanks!
>>
>> -- Emilio
>> _______________________________________________
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
signature.asc
Description: OpenPGP digital signature
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
