On Friday 17 June 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Thu, Jun 16, 2011 at 10:00:31AM CEST:
> > This refactoring should cause no API of functionality change,
> > and is meant only to simplify the future implementation of TAP
> > and SubUnit testsuite drivers. More precisely, our roadmap is
> > to move most of the "testsuite driving" features out of the
> > Automake-generated Makefiles, and into external scripts with
> > well-defined interfaces. This will allow the user to define
> > its own personalized testsuite drivers, and will also offer us
> > a framework upon which to implement our new TAP and SubUnit
> > drivers, all in a very unobtrusive way and retaining an high
> > degree of code reuse and backward-compatibility.
>
> I generally like the direction this is taking. The point of best
> separation between which code goes into Makefile.in and which into
> the driver scripts can be fine-tuned when we have more than one such
> script.
>
> Actually, yes, before deciding on this for real I really do want to see
> a nontrivial other driver script. There is no point in hardcoding
> too much in several driver scripts if it all needs to be the same
> anyway.
>
> Please measure the time overhead your changes introduce into the current
> code, for a trivial testsuite (say, 50 tests running 'true'), and a
> nontrivial one like Automake's and one with faster tests. Thanks.
>
Done.
I've run each of the testsuites repeated times, in the hope of shaking
out most of the noise.
I've run the Automake testsuite four times on a fast Solaris 10 system
(quadcore 3 GHz, 20 Gb of RAM) in low priority (a niceness of 19) and
with 32 processes concurrently ("make -j 32 check")
Then, I've run the Coreutils testsuite three times, on my Debian desktop
(single core 1.5 Ghz, 768 Mb of RAM), with one job at the time (i.e.,
plain "make check", no `-j' option).
Finally, I've run the "dummy" testsuite on my desktop ten times -- five
in serial mode ("make check") and five in concurrent mode, with four jobs
at the time ("make -j4 check").
-*-*-*-
Here are the results:
Automake Testsuite
==================
master branch
-------------
real 76m46.661s
user 127m54.170s
sys 136m31.526s
test-harness branch
-------------------
real 77m42.103s
user 126m7.108s
sys 140m26.200s
Coreutils Testsuite
===================
master branch
-------------
real 34m55.552s
user 4m23.216s
sys 6m20.312s
test-harness branch
-------------------
real 35m42.560s
user 4m29.453s
sys 6m27.740s
Dummy Testsuite
===============
master branch
-------------
real 1m3.158s
user 0m11.045s
sys 0m17.421s
test-harness branch
-------------------
real 1m11.025s
user 0m16.249s
sys 0m21.073s
-*-*-*-
Regads,
Stefano
