[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Luc Derrien updated SUREFIRE-1264:
---------------------------------------
Description:
Hello,
It appears some tests can be lost when using the parallel mode with
parameterized tests. Here is a [small
project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
used to try to reproduce the issue we have seen.
This is not something that happens on every run, but it's quite frequent.
With this loop, the problem should appear in less than 2 minutes, maybe on the
first run when (un)lucky:
{code}
time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
"$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
{code}
Normal run:
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
sleeptime = 1 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
sleeptime = 1 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
sleeptime = 2 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
sleeptime = 5 => stop
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - in
[p2]
Running [p2]
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in
[p2]
Results :
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
{noformat}
When tests have been lost (here one test lost according to the output):
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
sleeptime = 1 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
sleeptime = 2 => start
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in
[p2]
Running [p2]
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in
[p2]
Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
{noformat}
Note: there are 3 test classes and 18 tests on the [master
branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
even more frequent/easy to reproduce.
was:
Hello,
It appears some tests can be lost when using the parallel mode. Here is a
[small project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1]
I've used to try to reproduce the issue we have seen.
This is not something that happens on every run, but it's quite frequent.
With this loop, the problem should appear in less than 2 minutes, maybe on the
first run when (un)lucky:
{code}
time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
"$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
{code}
Normal run:
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
sleeptime = 1 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
sleeptime = 1 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
sleeptime = 2 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
sleeptime = 2 => stop
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
sleeptime = 5 => stop
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - in
[p2]
Running [p2]
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in
[p2]
Results :
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
{noformat}
When tests have been lost (here one test lost according to the output):
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running [p2]
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
sleeptime = 5 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
sleeptime = 1 => start
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
sleeptime = 2 => start
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in
[p2]
Running [p2]
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in
[p2]
Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
{noformat}
Note: there are 3 test classes and 18 tests on the [master
branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
even more frequent/easy to reproduce.
> Some tests can be lost when running in parallel with parameterized tests
> ------------------------------------------------------------------------
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
> Reporter: Jean-Luc Derrien
>
> Hello,
> It appears some tests can be lost when using the parallel mode with
> parameterized tests. Here is a [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec -
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
> even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
