On 12/03/15 22:51, Jonathan Griffin wrote: > The A-Team is embarking on a project to improve the developer experience > when running unittests locally. This project will address the following > frequently-heard complaints: > > * Locally developers often use mach to run tests, but tests in CI use > mozharness, which can result in different behaviors. > * It's hard to reproduce a try job because it's hard to set up the test > environment and difficult to figure out which command-line arguments to use. > * It's difficult to run tests from a tests.zip package if you don't have a > build on that machine and thus can't use mach. > * It's difficult to run tests through a debugger using a downloaded build.
So my problems might be slightly unusual because I am often working on the test harnesses themselves rather than the browser code. But a fairly common scenario I have is that tests fail on try in some platform I don't have access to locally (typically Windows). At this point I usually loan a slave and try to debug on there. The main problems I have are: * Getting mozharness to run in almost the same way as production, but without the bits that require it to actually be running in production. There is documentation for this, but it's still far from simple. * After reproducing the problems using the above setup it's usually necessary to add some logging, or other changes, to the tests or the harness. But it usually takes a couple of attempts to work out how to get mozharness to not overwrite my edited files with a freshly downloaded copy. * Once I've done this, if I want to actually land my changes I need to manually move them over from the unpacked version of tests.zip created by mozharness to a source tree and start a whole new try run that will go through a whole build cycle even if nothing in the browser itself has changed. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
