IME panels/popups/context-menus are flaky generally, and especially on
Linux. I don't know how to meaningfully lint for this.
The other thing is various incantations that try to do variations on the
"load x in a tab" theme. It turns out that it's not trivial to not race
with an expectant page load, especially if you don't have full control
over triggering it (ie the test is not the callsite of loadURI or
equivalent) and they involve new tabs, and e10s is on as well. Worse if
you also need to avoid hitting the network (see e.g.
waitForDocLoadAndStopIt for a demonstration of this).
Finally, the most common description of orange I see is some variation
of "test timed out". It would be helpful if we had more fluid rules
about test durations, so that mostly-unrelated patches don't "suddenly"
cause tests to cross the magical 45/90 second barrier (more often) and
then get backed out.
~ Gijs
On 06/07/2017 19:47, Andrew Halberstadt wrote:
I'm looking into creating custom eslint rules that aim to avoid common
causes of oranges in tests. We have an MDN page
<https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding_intermittent_oranges>
containing some of these already. Some of those patterns might be
pretty hard to catch with a linter, but others look do-able. So far,
I'm thinking of adding rules for:
* No arbitrary setTimeouts
* No synthesizeKey without waiting for focus
* No deleting original tab
* ???
These rules would only be configured to run on test files. I have a
few questions:
1. Would this cause too much of a burden for legitimate uses of those
patterns?
2. Do any of the other patterns on that page look feasible to
implement as linters?
3. Are there additional things not listed on that page that we could
lint for?
I'd love to find volunteer(s) who are more familiar with writing
mochitests than myself (and who also care about this issue) to help
brainstorm and provide more in-depth feedback. I want to strike a
balance between preventing oranges and not over-burdening test
authors, but I'm not sure what that balance looks like.
Thanks in advance for your input,
Andrew
_______________________________________________
firefox-dev mailing list
[email protected]
https://mail.mozilla.org/listinfo/firefox-dev
_______________________________________________
dev-platform mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-platform
