https://bugs.kde.org/show_bug.cgi?id=502679
--- Comment #15 from Mark Wielaard <m...@klomp.org> ---
Note again that this is in an already closed bug, it would be better to just
open a new bug so we don't loose this.
But we are close. Just some small comments.
(In reply to mcermak from comment #14)
> Created attachment 180638 [details]
> parallelize testing
>
> Re-format the previous patch using git format-patch to standardize it.
> Please, review.
+PARALLEL_JOBS=${PARALLEL_JOBS:-$(($(nproc --all) + 10))}
Could we pass this from the Makefile instead? So that it follows the make -jx
ltpchecks from top-level?
I am thinking of something like:
diff --git a/auxprogs/Makefile.am b/auxprogs/Makefile.am
index 9cec4f222b36..3a2d0d17629e 100644
--- a/auxprogs/Makefile.am
+++ b/auxprogs/Makefile.am
@@ -231,7 +231,13 @@ gsl-check: $(GSL_BUILD_DIR)/gsl-build
diff -u $(abs_top_srcdir)/auxprogs/gsl-1.6.out.x86.exp \
$(GSL_BUILD_DIR)/valgrind-gsl.out
+# Extract -jNUM from MAKEFLAGS.
+# Note that any spaces between -j and NUM have already been removed.
+# Also there will be only one (the last) -jNUM left in MAKEFLAGS.
+PARALLEL_JOBS=$(subst -j,,$(filter -j%,$(MAKEFLAGS)))
+
ltp-check: $(LTP_SRC_DIR)
+ PARALLEL_JOBS=$(PARALLEL_JOBS) \
LTP_SRC_DIR=$(LTP_SRC_DIR) \
VALGRIND=$(abs_top_builddir)/vg-in-place \
$(abs_top_srcdir)/auxprogs/ltp-tester.sh
Also could we use just $(nproc) instead of --all and +10. I am afraid to
overload a machine otherwise.
With the above, if people really want to use that then they can use make -j
$[$(nproc --all) + 10]
+ while test "$(pgrep -P $$ | wc -l)" -gt $PARALLEL_JOBS; do sleep 0.1; done
I think jobs -l is slightly more efficient than pgrep -P $$
The sleep 0.1 while loop isn't great, but I don't know a good alternative.
Don't you have to pass $test and $i to doTest?
It seems they could change before they are used in a particular doTest run.
Maybe something like:
diff --git a/auxprogs/ltp-tester.sh b/auxprogs/ltp-tester.sh
index 412a1ac55089..e31f8a6b12e4 100755
--- a/auxprogs/ltp-tester.sh
+++ b/auxprogs/ltp-tester.sh
@@ -14,7 +14,7 @@ SUMMARY_LOG="$LOGDIR/summary.log"
DIFFCMD="diff -u"
VALGRIND="${VALGRIND:-$LTP_SRC_DIR/../../../vg-in-place}"
# For parallel testing, consider IO intensive jobs, take nproc into account
-PARALLEL_JOBS=${PARALLEL_JOBS:-$(($(nproc --all) + 10))}
+PARALLEL_JOBS=${PARALLEL_JOBS:-$(nproc)}
# TESTS env var may be specified to restrict testing to selected test cases
# Initialize LOGDIR
@@ -29,11 +29,13 @@ myLog ()
doTest ()
{
- dir=$(dirname $test)
- exe=$(basename $test)
+ t=$1
+ nr=$2
+ dir=$(dirname $t)
+ exe=$(basename $t)
l="$LOGDIR/$exe"
mkdir -p $l
- echo "[$i/$c] Testing $exe ..." | tee -a $l/summary
+ echo "[$nr/$c] Testing $exe ..." | tee -a $l/summary
pushd $dir >/dev/null
PATH="$ORIG_PATH:$PWD"
./$exe >$l/log1std 2>$l/log1err ||:
@@ -88,9 +90,9 @@ c=${#files[@]}; i=0
# Run tests in parallel
for test in "${files[@]}"; do
- while test "$(pgrep -P $$ | wc -l)" -gt $PARALLEL_JOBS; do sleep 0.1; done
+ while test "$(jobs -l | wc -l)" -gt $PARALLEL_JOBS; do sleep 0.1; done
i=$((++i))
- doTest &
+ doTest $test $i &
done
wait
--
You are receiving this mail because:
You are watching all bug changes.
