Re: Running tests from the source checkout

2016-09-23 Thread Aki Sasaki
If we compile locally before testing in automation, we're no longer testing the same bits that we're going to be shipping. That's a major change in direction, and one that may have significant consequences. An additional historical reason for keeping development tools off test machines: we once s

Re: Running tests from the source checkout

2016-09-23 Thread Dustin Mitchell
Worth noting that, at least for a while, our desktop-test Docker images did include a compiler for other purposes, so the library-dependency risk is already present. Explicit checks for that seem like the right approach. I believe we have a few (around glibc symbols, at least) but perhaps not com

Re: Running tests from the source checkout

2016-09-22 Thread Gregory Szorc
> On Sep 22, 2016, at 17:27, Aki Sasaki wrote: > > If we compile locally before testing in automation, we're no longer testing > the same bits that we're going to be shipping. That's a major change in > direction, and one that may have significant consequences. We would only compile tests i

Re: Running tests from the source checkout

2016-09-22 Thread Gregory Szorc
On Thu, Sep 22, 2016 at 2:49 PM, Nicholas Alexander wrote: > > > On Thu, Sep 22, 2016 at 1:20 PM, Gregory Szorc wrote: > >> For years it has been a common pattern for local builds and Firefox >> automation to copy test files to the object directory (or some other >> staging area) and run them fr

Re: Running tests from the source checkout

2016-09-22 Thread Nicholas Alexander
On Thu, Sep 22, 2016 at 1:20 PM, Gregory Szorc wrote: > For years it has been a common pattern for local builds and Firefox > automation to copy test files to the object directory (or some other > staging area) and run them from there. This creates several inefficiencies: > > * The build system m