On 6/12/14, 2:07 PM, Armen Zambrano G. wrote:
Hello all,
I've recently been working on adding disabled tests into our tests.zip.
There are some challenges that I have faced and I wonder what I ideas
others have to improve it.
(I'm explicitly not going to go into a blue sky scenario where tests.zip
are not needed to begin with. If you prefer we can fork that thread as
it would specifically help the releng integration systems: less overhead
& network usage)
== Local changes versus source code changes ==
For local testing, I make changes on the files out of the tests.zip and
then remember to put it back into my source tree.
If I'm not careful, the changes can easily be lost or never make it back
into the source tree.
== Faster tests.zip generation ==
Is there a faster way to generate a tests.zip than ./mach build and make
package-tests from the objdir?
For instance, generating a tests.zip for b2g emulators requires going
through the whole setup of making b2g emulator builds (I'm still
building atm).
In my case, I cheated the system by downloading a tests.zip file to
people.mozilla.com, hack it, zip it back and trigger an arbitrary job.
If I start from a tests.zip and work from there (unless I need to create
new binaries) I think I'm OK.
I think this would help me, a .mach command that would:
* download a zip file for a platform
* unzip it
* put my changes from the source code into it
* zip it back
What do people think?
== Multiple platform ==
I've noticed that from platform to platform the tests.zip are slightly
different. They can contain these:
* binaries
* runners
* tests
* manifests
* anything else?
In my development, it would have been awesome to generate a tests.zip
files that would be able to run on any platform. Unfortunately, the
binaries are coupled and test packaging is tied to the host packing
operating system.
(Adding dev-builds)
We've talked about improving test archive generation from build system
land. The current solution is a giant hack and inefficient. Ted has some
ideas and a bug on file (though I can't find it ATM). We should be able
to get to a state that doesn't require a build to generate a tests archive.
Given our new ability to trigger test jobs against arbitrary test
archives, I think having the ability to create one-off test archives for
testing is a terrific idea. I also support the idea of unified tests
archives. Although, that may require changing our moz.build parsing
strategy to include parsing every single file, despite the current build
configuration. We've considered that before, so it's not a crazy idea.
Your ideas are terrific. I just don't know if the resources are there to
prioritize this.
As a shim, we could probably provide a mach command to package tests and
optionally overlay in-tree test files (at least the one moz.build knows
about - test harness files are currently notably absent) over an
existing tests archive.
_______________________________________________
dev-builds mailing list
dev-builds@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-builds
