On 11/06/2013 03:58 AM, Aryeh Gregor wrote:
> Has anyone considered allowing try pushes to run only specified
> directories of tests, and to allow incremental builds rather than
> clobbers on try? This would make try a heck of lot faster and
> resource-efficient, for those who are willing to accept less certain
> results.
I think this is a good idea for the "faster" reason. I think it's a bad
idea for the "resource-efficient" reason.
Trychooser's test suite granularity is already overused from the
resource consumption perspective. It saves relatively few resources to
select eg mochitest-3 instead of just running all the mochitests,
especially as compared to the cost of a backout from a failure in one of
the tests that wasn't run. A |-u mochitests| run is already way cheaper
than a |-u all| run, and we'll always have lots of |-u all| pushes going
on, so you're on the wrong side of Amdahl's Law. Whittling an individual
test suite down to fewer tests is even less worthwhile.
As I've said before, you're better off doing a T push. I just looked at
a random try push, and it has *16* full sets of tests on desktop
(linux32, linux64, 3 x Windows, 3 x OSX) x (opt, debug). Doing |-u
all[x64]| is 12.5% of the load, and most test failures aren't
platform-dependent. (Build failures are.) And personally, I really like
having the full set of tests, because I often break both a mochitest and
an xpcshell test, and I'd much rather debug an xpcshell failure than a
mochitest failure. So you're not just pushing to get a single bit back
("did I break anything?"), you're also pushing to get back "what's the
simplest thing I broke?"
As for "faster", I'm skeptical of the reliability of incremental try
builds. We have too many clobbers. And the same slaves do a bunch of
different build types, so it may have been quite a while since the last
build of the type you're doing. (Sure, we could tweak the scheduling to
add some affinity in, but that's more complexity and a richer set of
failure patterns.) I'm still a bit curious as to whether allowing slaves
to collaborate via a fast distcc network going to remote ccaches would
work, but it also feels complex and potentially counterproductive (I'm
not at all sure that it's faster to look up and transfer a remotely
cached build result than it would be to recompile it locally.)
Running subsets of the tests seems useful for latency, though. I mean,
you can already do that with manifests or nuking tests or something on a
try push, but it's painful enough that I rarely do it. And the current
try server buildbot overhead is too high for this to be truly useful,
though. We'd need a lighter-weight -- or at least, faster -- mechanism
for "rerun on just these tests" that would produce results in about the
time it takes to run them. That would be really nice. And maybe you
could use incremental builds after all, if you could say "I'm gonna be
hacking on this stuff for a bit" and it autoreserved you a slave or
copied over your bits to a dedicated box.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
