FWIW years ago I decided to act on trying to detect some of the patterns
indicated on that page automatically, see
https://bugzilla.mozilla.org/show_bug.cgi?id=649012. It didn't use
linting (because we didn't really have any linting support back then)
but the idea is the same, just with a different syntax. That project
never really received much (well, any!) traction since I always was
hoping that people would triage the initial set of "opt-out" annotations
that it landed with, and do something about those tests, but that never
happened. But hopefully this helped a bit by plugging one of the holes
in terms of the unintended sources of intermittent oranges.
Based on that experience, here are some answers:
On 07/06/2017 02:47 PM, Andrew Halberstadt wrote:
I'm looking into creating custom eslint rules that aim to avoid common
causes of oranges in tests. We have an MDN page
<https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding_intermittent_oranges>
containing some of these already. Some of those patterns might be
pretty hard to catch with a linter, but others look do-able. So far,
I'm thinking of adding rules for:
* No arbitrary setTimeouts
* No synthesizeKey without waiting for focus
* No deleting original tab
* ???
These rules would only be configured to run on test files. I have a
few questions:
1. Would this cause too much of a burden for legitimate uses of those
patterns?
In this case the way to opt out was adding a single line to the test.
This wasn't hard per se, but if my memory serves it used to trip people
for quite a while, even though the failure message you would get if you
forgot to do it would indicate how to opt out of the extra check. In
hindsight, I think some people may have found the check more annoying
than helpful, because in the cases where it helps, you'd get green
tests, so nobody would consciously know that this is helping and in the
cases where it would get in your way, people would be rightfully unhappy
since their workflow got disturbed.
1. Do any of the other patterns on that page look feasible to
implement as linters?
I always struggled with this one. Not many of them are, in my opinion.
The example of using Date()s is, for example, but in my experience that
is a super rare cause of intermittent failures... A lot of these
require some kind of dynamic check at runtime, and aren't easily
detectible by just looking at the code of the test without running it.
1. Are there additional things not listed on that page that we could
lint for?
Almost certainly. Ideally when people fix an intermittent orange, they
would think about whether this was a pattern that could reappear in
other places, and whether it could be detected using a linter. It is
unclear to me how many of these new unique patterns we encounter
though. (For example, *many* intermittent oranges boil down to code
expecting the wrong ordering of events, and that's not easily detectible
using a linter.... But there may be a long tail of super rare error
cases that are detectible using linters.)
Hope this helps,
Ehsan
I'd love to find volunteer(s) who are more familiar with writing
mochitests than myself (and who also care about this issue) to help
brainstorm and provide more in-depth feedback. I want to strike a
balance between preventing oranges and not over-burdening test
authors, but I'm not sure what that balance looks like.
Thanks in advance for your input,
Andrew
_______________________________________________
firefox-dev mailing list
firefox-...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
