While waiting on testsuites this week, I finally snapped and spent some
time looking at how to speed up the testsuite.
I did some experiments and collected data on the normalised runtimes of
each .exp test script. I sorted them in descending order and these are
the top offenders:
normalised
dg.exp,gfortran
0.283
execute.exp,gcc
0.142
compile.exp,gcc
0.102
execute.exp,gfortran
0.046
struct-layout-1.exp,g++
0.040
struct-layout-1.exp,gcc
0.037
dg.exp,g++
0.037
builtins.exp,gcc
0.036
One problem with parallelising the gcc testsuite is that many of the
test drivers are not parallel safe. So, rather than try to make them
all parallel safe, I decided we should just pick off the worst offenders
and manually schedule these (for all languages): run the big .exp
scripts in parallel with "the rest". In the list above, for example,
gfortran and g++ dg.exp are run in parallel, despite belonging to
different languages. For added robustness, each of the big test driver
is given its own testsuite directory (as we do now for checking
languages in parallel).
I ran as many of the top offenders in parallel as required to get "the
rest" to require the same normalised time as gfortran dg.exp, as we
cannot go any faster than the longest running .exp script. I prototyped
this using the following shell script, which drives the make checks:
#!/bin/sh
# Run these in parallel:
tests="gfortran:dg.exp gcc:execute.exp gcc:compile.exp gfortran:execute.exp
gcc:struct-layout-1.exp g++:struct-layout-1.exp g++:dg.exp gcc:builtins.exp
gcc:dg.exp gcc:dg-torture.exp"
if [ ! -f auto-host.h ] ; then
echo "not a gcc build directory"
exit 1
fi
make testsuite/site.exp
for item in $tests ; do
make RUNTESTFLAGS="--tool ${item%:*} ${item#*:}" check-${item%:*} &
done
# Run the rest of the tests, but ignore the ones we've already run.
# Run all languages in parallel.
for tool in gcc g++ gfortran objc ; do
ignore=""
for item in $tests ; do
if [ ${item%:*} = ${tool} ] ; then
ignore="$ignore ${item#*:}"
fi
done
make RUNTESTFLAGS="--tool ${tool} --ignore $ignore" check-${tool} &
done
wait
Using this script and some minor gcc/Makefile.in hacks, I ran the entire
testsuite in 30% of the current time for a parallel-languages make
check.
So, I guess my question is: what now? What do people feel would be
required to make this usable? I assume that the most pressing thing
would be to have the build system fold the various .log and .sum files
together so that they look like they were run as a whole.
Another problem is that this is a bit of guesswork. Maybe it would be
better to drive this from make -j, so that make takes care of the
scheduling (and we can push the longer tests to the front of the
dependency list, as we do to improve parallel builds now).
Comments?
Cheers, Ben
