[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064858#comment-16064858
]
Tanguy Le Meur commented on SUREFIRE-1264:
------------------------------------------
Hi [~tibor17],
Hmmm, that's surprising that you don't reproduce, whilst it's a bit random, I
reproduce it more than 75% of the time (so waiting a few execution only show
the problem rapidly) - <parallel>all</parallel> helps a lot here to make it
happen more frequently.
I fixed the POM in the branch as you suggested. I still reproduce.
Here is a trace of a failing case:
{code}
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building main 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ main ---
[INFO] Deleting /Users/tlemeur/git/surefire-junit-tests/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ main ---
[WARNING] Using platform encoding (US-ASCII actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/Users/tlemeur/git/surefire-junit-tests/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ main ---
[WARNING] File encoding has not been set, using platform encoding US-ASCII,
i.e. build is platform dependent!
[INFO] Compiling 1 source file to
/Users/tlemeur/git/surefire-junit-tests/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
main ---
[WARNING] Using platform encoding (US-ASCII actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/Users/tlemeur/git/surefire-junit-tests/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ main
---
[WARNING] File encoding has not been set, using platform encoding US-ASCII,
i.e. build is platform dependent!
[INFO] Compiling 4 source files to
/Users/tlemeur/git/surefire-junit-tests/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.21-SNAPSHOT:test (default-test) @ main ---
[WARNING] Missing POM for
org.apache.maven.surefire:surefire-booter:jar:2.21-SNAPSHOT
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.ATest
Start: ATest.methodA1[p0]
End: ATest.methodA1[p0]
Start: ATest.methodA1[p1]
End: ATest.methodA1[p1]
Start: ATest.methodA2[p0]
Start: ATest.methodA2[p1]
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.112 s
<<< FAILURE! - in com.appnexus.viewability.core.surefireJunitTests.ATest
[ERROR] methodA1[p0](com.appnexus.viewability.core.surefireJunitTests.ATest)
Time elapsed: 0.112 s <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p0]
at
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)
[ERROR] methodA1[p1](com.appnexus.viewability.core.surefireJunitTests.ATest)
Time elapsed: 0.112 s <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p1]
at
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.CTest
Start: CTest.methodC1[p0]
End: CTest.methodC1[p0]
Start: CTest.methodC2[p0]
End: CTest.methodC2[p0]
Start: CTest.methodC1[p1]
End: CTest.methodC1[p1]
Start: CTest.methodC2[p1]
End: CTest.methodC2[p1]
Start: CTest.methodC1[p2]
End: CTest.methodC1[p2]
Start: CTest.methodC2[p2]
End: CTest.methodC2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.524 s
- in com.appnexus.viewability.core.surefireJunitTests.CTest
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.BTest
Start: BTest.methodB2[p0]
End: BTest.methodB2[p0]
Start: BTest.methodB1[p1]
End: BTest.methodB1[p1]
Start: BTest.methodB1[p0]
End: BTest.methodB1[p0]
Start: BTest.methodB2[p1]
End: BTest.methodB2[p1]
Start: BTest.methodB1[p2]
End: BTest.methodB1[p2]
Start: BTest.methodB2[p2]
End: BTest.methodB2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.312 s
- in com.appnexus.viewability.core.surefireJunitTests.BTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ATest.methodA1:14 Failing test: ATest.methodA1[p0]
[ERROR] ATest.methodA1:14 Failing test: ATest.methodA1[p1]
[INFO]
[ERROR] Tests run: 16, Failures: 2, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.683 s
[INFO] Finished at: 2017-06-27T15:33:49+02:00
[INFO] Final Memory: 17M/169M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.21-SNAPSHOT:test
(default-test) on project main: There are test failures.
[ERROR]
[ERROR] Please refer to
/Users/tlemeur/git/surefire-junit-tests/target/surefire-reports for the
individual test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump,
[date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
{code}
As you can see, the tests started with "ATest" in the trace, in this case, it's
wrong. In your case it did CTest, ATest then BTest (when this order happen for
me, it works fine too). The differences between our execution looks to be here.
For the needs of this demo, I'm not sure how to have the test to run in a given
order... I tried <runOrder> but it seems not to impact the actual order.
I would be happy to spend some time helping you on this, I'm just not really
sure how to help. Ideally I'd like to make it 100% of the time reproductible.
> Some tests can be lost when running in parallel with parameterized tests
> ------------------------------------------------------------------------
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
> Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
> Reporter: Jean-Luc Derrien
> Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with
> parameterized tests. Here is a [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec -
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
> even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
