If we compile locally before testing in automation, we're no longer testing the same bits that we're going to be shipping. That's a major change in direction, and one that may have significant consequences.
An additional historical reason for keeping development tools off test machines: we once shipped a new runtime dependency that we didn't detect because the test machines all had development environments. Users on that platform were unable to run Firefox without installing a 3rd party library. I very much agree with the goal of simplifying and streamlining how we run tests. I think we do have to keep the above in mind when we go about that simplification. On Thu, Sep 22, 2016 at 3:54 PM, Gregory Szorc <g...@mozilla.com> wrote: > On Thu, Sep 22, 2016 at 2:49 PM, Nicholas Alexander < > nalexan...@mozilla.com> > wrote: > > > > > > > On Thu, Sep 22, 2016 at 1:20 PM, Gregory Szorc <g...@mozilla.com> wrote: > > > >> For years it has been a common pattern for local builds and Firefox > >> automation to copy test files to the object directory (or some other > >> staging area) and run them from there. This creates several > inefficiencies: > >> > >> * The build system must symlink or copy thousands of test related files > >> during the build > >> * Running tests requires checking if test files in the objdir are up to > >> date (can add significant latency to the edit-test loop) > >> * Builds in automation must create and upload archives containing > >> thousands of test files > >> * This makes it harder to skip build jobs if all that has changed is a > >> test file > >> * Test jobs in automation must download and extract archives containing > >> test files > >> * Test jobs in automation typically don't have access to a full source > >> checkout and can't easily piggyback on existing infrastructure (such as > >> mach commands, vendored Python packages, etc). This leads to wheel > >> reinvention and inconsistency between local dev environments and > automation. > >> > >> The historical reasons for doing things this way were valid. But with > >> moz.build, other improvements to the build system, automation scheduling > >> and configuration living in-tree, and better scaling of Mercurial in > >> automation, these historical reasons are largely no longer valid. > >> > >> Going forward, I'm requesting we change our policy regarding tests and > >> automation to be source checkout first. This means: > >> > >> * Tests should be designed to run from a source checkout with minimal > >> "build" actions required (no file copying/symlinking, no preprocessing > into > >> a new file, etc) > >> * Automation should run everything from a source checkout (as opposed to > >> downloading archives containing files derived from source checkouts) > >> > >> Does anyone have any concerns or objections to this? > >> > > > > This only makes sense for a subset of our harnesses. Tests that need to > > be compiled (gtest?, Android unit and Robocop tests) can never achieve > > this. So as long as we're clear that we want to remove preprocessing and > > special mozharness-foo to work around packaging tests but we're not > pushing > > *everything* to source-only, I'm good. > > > > The goal is to reduce the amount of work needed to run tests. You are > correct that compiled tests inherently need a build step to run them and we > can't simply run them from a source checkout. > > That being said, it isn't out of the question to defer the compilation of > these tests to test jobs: the build job delays execution of nearly all > tests and making the build job faster by moving work out of that job could > make end-to-end times faster, even if we're doing extra work in individual > test jobs (that have to compile tests). Historically, we didn't have a > compiler available on test machines. But with TaskCluster+Docker and > tooltool, it is much easier to put needed files on testers. So compiling > tests on test machines is something we should consider. > _______________________________________________ > tools mailing list > to...@lists.mozilla.org > https://lists.mozilla.org/listinfo/tools >
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
