IME the issue is not so much about not running tests identical to the
ones on CI, but the OS environment which doesn't match, and then
reproducing intermittent failures.
If a failure happens once in 100 builds, it is very annoying for the
sheriffs (happens multiple times a day) and needs fixing or backing out
- but running-by-dir, say, mochitest-browser for
browser/base/content/test/general/ 100 times takes way too long, and OS
settings / screen sizes / machine speed / (...) differences mean you
might not be able to reproduce anyway (or in the worst case, that you
get completely different failures).
It'd be better if we could more easily get more information about
failures as they happened on infra (replay debugging stuff a la what roc
has worked on, or better logs, or somehow making it possible to
remote-debug the infra machines as/when they fail).
~ Gijs
On 12/03/2015 22:51, Jonathan Griffin wrote:
The A-Team is embarking on a project to improve the developer experience
when running unittests locally. This project will address the following
frequently-heard complaints:
* Locally developers often use mach to run tests, but tests in CI use
mozharness, which can result in different behaviors.
* It's hard to reproduce a try job because it's hard to set up the test
environment and difficult to figure out which command-line arguments to use.
* It's difficult to run tests from a tests.zip package if you don't have a
build on that machine and thus can't use mach.
* It's difficult to run tests through a debugger using a downloaded build.
The quintessential use case here is making it easy to reproduce a try run
locally, without a local build, using a syntax something like:
* runtests --try 2844bc3a9227
Ideally, this would download the appropriate build and tests.zip package,
bootstrap the test environment, and run the tests using the exact same
arguments as are used on try, optionally running it through an appropriate
debugger. You would be able to substitute a local build and/or local
tests.zip package if desired. You would be able to override command-line
arguments used in CI if you wanted to, otherwise the tests would be run
using the same args as in CI.
What other use cases would you like us to address, which aren't derivatives
of the above issues?
Thanks for your input,
Jonathan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
