On Fri, Jul 6, 2012 at 6:02 PM, Stefano Lattarini
<stefano.lattar...@gmail.com> wrote:
> It turns out that, with NetBSD 5.1 make and FreeBSD 9 make, running
> "make recheck" two times in a row quickly fails to correctly re-run
> the failed tests in the second run.
>
> That issue has been introduced likely introduced in commit
That issue was likely introduced in...
> 'v1.12.1-95-gd5443e4' of 20102-07-01, "parallel-tests: reimplement
> fix for bug#11791".
>
> Anyway, the use case that has been broken is not realistic (who is
> going to run "make recheck" two times in one second, without modifying
> any of the tests or the tested programs in the meantime?), so we believe
> the best fix is to simply work around the issue in the affected test
> cases, rather than risking to slow down or uglify the 'recheck' rule.
rather than risk slowing down or uglifying the 'recheck' rule.
> * t/parallel-tests9.sh: Enhance a little.
> * t/tap-recheck.sh: Adjust adding proper '$sleep' calls were required.
s/were/where/
> * t/parallel-tests-log-override-recheck.sh: Likewise.
> * t/test-driver-custom-multitest-recheck.sh: Likewise.
> * t/test-driver-custom-multitest-recheck2.sh: Likewise.
>
> Signed-off-by: Stefano Lattarini <stefano.lattar...@gmail.com>
> ---
>
> Please scrap the previous patch. I don't know what I was thinking when
> I wrote it (assuming I was thinking at all, which I much doubt). Of
> course it solves the regression, but doing so re-introduces bug#11791!
> At least I did proper testing before pushing, so the error was caught
> in time.
>
> So, OK to apply? I will push shortly if there is no objection.
>
> Thanks, and sorry for the noise,
> Stefano
>
> t/parallel-tests-log-override-recheck.sh | 2 ++
> t/parallel-tests9.sh | 5 +++--
> t/tap-recheck.sh | 7 +++++++
> t/test-driver-custom-multitest-recheck.sh | 8 ++++++++
> t/test-driver-custom-multitest-recheck2.sh | 6 ++++++
> 5 files changed, 26 insertions(+), 2 deletions(-)
>
> diff --git a/t/parallel-tests-log-override-recheck.sh
> b/t/parallel-tests-log-override-recheck.sh
> index 493fbfe..79c16fc 100755
> --- a/t/parallel-tests-log-override-recheck.sh
> +++ b/t/parallel-tests-log-override-recheck.sh
> @@ -77,6 +77,8 @@ for x in stdout my.log; do
> $FGREP bazbaz $x
> done
>
> +using_gmake || $sleep # Required by BSD make.
> +
> chmod a-rw my.log
> BAZ_EXIT_STATUS=0 TEST_SUITE_LOG=my2.log $MAKE -e recheck >stdout \
> && { cat stdout; exit 1; }
> diff --git a/t/parallel-tests9.sh b/t/parallel-tests9.sh
> index 08d2bf6..ea9dba6 100755
> --- a/t/parallel-tests9.sh
> +++ b/t/parallel-tests9.sh
> @@ -59,9 +59,10 @@ $MAKE check >stdout && { cat stdout; exit 1; }
> cat stdout
> count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
>
> -# Running this two times in a row proved problematic on NetBSD make
> -# in the past. Ensure we don't regress in that respect.
> +# Running this two times in a row should produce the same results the
> +# second time.
> for i in 1 2; do
> + using_gmake || $sleep # Required by BSD make.
> $MAKE recheck >stdout && { cat stdout; exit 1; }
> cat stdout
> count_test_results total=2 pass=0 fail=1 skip=0 xfail=0 xpass=0 error=1
> diff --git a/t/tap-recheck.sh b/t/tap-recheck.sh
> index 877f70e..3c542ba 100755
> --- a/t/tap-recheck.sh
> +++ b/t/tap-recheck.sh
> @@ -130,6 +130,7 @@ for vpath in : false; do
> count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
>
> : Let us make b.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo OK > b.ok
> do_recheck --fail
> # a.test has been successful the first time, so no need to re-run it.
> @@ -141,6 +142,7 @@ for vpath in : false; do
> count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
>
> : Let us make the first part of c.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo OK > c.pass
> do_recheck --fail
> test ! -e a.run
> @@ -150,6 +152,7 @@ for vpath in : false; do
> count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
>
> : Let us make also the second part of c.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo KO > c.xfail
> do_recheck --fail
> test ! -e a.run
> @@ -160,6 +163,7 @@ for vpath in : false; do
>
> : Nothing changed, so only d.test should be run.
> for i in 1 2; do
> + using_gmake || $sleep # Required by BSD make.
> do_recheck --fail
> test ! -e a.run
> test ! -e b.run
> @@ -169,6 +173,7 @@ for vpath in : false; do
> done
>
> : Let us make d.test run more testcases, and experience _more_ failures.
> + using_gmake || $sleep # Required by BSD make.
> echo 'test_count=9' > d.count
> unindent > d.extra <<'END'
> echo 'ok # SKIP s'
> @@ -190,6 +195,7 @@ END
> count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
>
> : Let us finally make d.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo 'test_count=1' > d.count
> echo : > d.extra
> do_recheck --pass
> @@ -200,6 +206,7 @@ END
> count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
>
> : All tests have been successful or skipped, nothing should be re-run.
> + using_gmake || $sleep # Required by BSD make.
> do_recheck --pass
> test ! -e a.run
> test ! -e b.run
> diff --git a/t/test-driver-custom-multitest-recheck.sh
> b/t/test-driver-custom-multitest-recheck.sh
> index ae31397..362abbf 100755
> --- a/t/test-driver-custom-multitest-recheck.sh
> +++ b/t/test-driver-custom-multitest-recheck.sh
> @@ -107,6 +107,7 @@ for vpath in : false; do
> $srcdir/configure
>
> : A "make recheck" in a clean tree should run no tests.
> + using_gmake || $sleep # Required by BSD make.
> do_recheck --pass
> cat test-suite.log
> test ! -e a.run
> @@ -131,6 +132,7 @@ for vpath in : false; do
> count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
>
> : Let us make b.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo OK > b.ok
> do_recheck --fail
> # a.test has been successful the first time, so no need to re-run it.
> @@ -142,6 +144,7 @@ for vpath in : false; do
> count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
>
> : Let us make the first part of c.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo OK > c.pass
> do_recheck --fail
> test ! -e a.run
> @@ -151,6 +154,7 @@ for vpath in : false; do
> count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
>
> : Let us make also the second part of c.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo KO > c.xfail
> do_recheck --fail
> test ! -e a.run
> @@ -161,6 +165,7 @@ for vpath in : false; do
>
> : Nothing changed, so only d.test should be run.
> for i in 1 2; do
> + using_gmake || $sleep # Required by BSD make.
> do_recheck --fail
> test ! -e a.run
> test ! -e b.run
> @@ -170,6 +175,7 @@ for vpath in : false; do
> done
>
> : Let us make d.test run more testcases, and experience _more_ failures.
> + using_gmake || $sleep # Required by BSD make.
> unindent > d.extra <<'END'
> echo SKIP: s
> echo FAIL: f 1
> @@ -190,6 +196,7 @@ END
> count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
>
> : Let us finally make d.test pass.
> + using_gmake || $sleep # Required by BSD make.
> echo : > d.extra
> do_recheck --pass
> test ! -e a.run
> @@ -199,6 +206,7 @@ END
> count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
>
> : All tests have been successful or skipped, nothing should be re-run.
> + using_gmake || $sleep # Required by BSD make.
> do_recheck --pass
> test ! -e a.run
> test ! -e b.run
> diff --git a/t/test-driver-custom-multitest-recheck2.sh
> b/t/test-driver-custom-multitest-recheck2.sh
> index 623cbad..b6f543a 100755
> --- a/t/test-driver-custom-multitest-recheck2.sh
> +++ b/t/test-driver-custom-multitest-recheck2.sh
> @@ -105,6 +105,7 @@ for vpath in : false; do
> unset var
>
> : a.test was successful the first time, no need to re-run it.
> + using_gmake || $sleep # Required by BSD make.
> env TESTS=a.test $MAKE -e recheck >stdout \
> || { cat stdout; exit 1; }
> cat stdout
> @@ -114,6 +115,7 @@ for vpath in : false; do
> test ! -e c.run
>
> : b.test failed, it should be re-run. And make it pass this time.
> + using_gmake || $sleep # Required by BSD make.
> echo OK > b.ok
> TEST_LOGS=b.log $MAKE -e recheck >stdout \
> || { cat stdout; exit 1; }
> @@ -126,6 +128,7 @@ for vpath in : false; do
> rm -f *.run
>
> : No need to re-run a.test or b.test anymore.
> + using_gmake || $sleep # Required by BSD make.
> TEST_LOGS=b.log $MAKE -e recheck >stdout \
> || { cat stdout; exit 1; }
> cat stdout
> @@ -133,6 +136,7 @@ for vpath in : false; do
> test ! -e a.run
> test ! -e b.run
> test ! -e c.run
> + using_gmake || $sleep # Required by BSD make.
> TESTS='a.test b.test' $MAKE -e recheck >stdout \
> || { cat stdout; exit 1; }
> cat stdout
> @@ -158,6 +162,7 @@ for vpath in : false; do
> : c.test contained and hard error the last time, so it should be re-run.
> : This time, make it pass
> # Use 'echo', not ':'; see comments above for why.
> + using_gmake || $sleep # Required by BSD make.
> echo dummy > c.ok
> env TESTS='c.test a.test' $MAKE -e recheck >stdout \
> || { cat stdout; exit 1; }
> @@ -171,6 +176,7 @@ for vpath in : false; do
>
> : Nothing should be rerun anymore, as all tests have been eventually
> : successful.
> + using_gmake || $sleep # Required by BSD make.
> $MAKE recheck >stdout || { cat stdout; exit 1; }
> cat stdout
> count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
> --
> 1.7.9.5
>
>
Cheers,
Dave Hart
