[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061108#comment-16061108
]
Tanguy Le Meur commented on SUREFIRE-1264:
--
Hi [~tibor17],
Thank a lot for your fix. I did some extensive testing on the 2.21-SNAPSHOT
version, and this indeed improves a lot. For example, I was not able to
reproduce the error using [~jderrien] branch.
However if a test fails (Assert.fail), later tests can still be lost.
I made a branch here to show the problem:
https://github.com/tanguylemeur/surefire-junit-tests/tree/failure-2.21-snapshot-SUREFIRE-1264
If it would be ok, then this command should never end:
{code}time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
"$(grep -c 'Tests run: 18' last.log)" == "0" ]; then break; fi ; done{code}
Actually if you look at the trace, it seems that the test count varies 100% of
the time if it starts with the class "ATest". As far I understand, after a test
fails, all tests from this class are not executed anymore.
Let me know if you need more information, or if you want me to open a separate
bug?
Thank you!
> Some tests can be lost when running in parallel with parameterized tests
>
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with
> parameterized tests. Here is a [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec -
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
> even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064858#comment-16064858
]
Tanguy Le Meur commented on SUREFIRE-1264:
--
Hi [~tibor17],
Hmmm, that's surprising that you don't reproduce, whilst it's a bit random, I
reproduce it more than 75% of the time (so waiting a few execution only show
the problem rapidly) - all helps a lot here to make it
happen more frequently.
I fixed the POM in the branch as you suggested. I still reproduce.
Here is a trace of a failing case:
{code}
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building main 1.0.0
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ main ---
[INFO] Deleting /Users/tlemeur/git/surefire-junit-tests/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ main ---
[WARNING] Using platform encoding (US-ASCII actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/Users/tlemeur/git/surefire-junit-tests/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ main ---
[WARNING] File encoding has not been set, using platform encoding US-ASCII,
i.e. build is platform dependent!
[INFO] Compiling 1 source file to
/Users/tlemeur/git/surefire-junit-tests/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
main ---
[WARNING] Using platform encoding (US-ASCII actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
/Users/tlemeur/git/surefire-junit-tests/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ main
---
[WARNING] File encoding has not been set, using platform encoding US-ASCII,
i.e. build is platform dependent!
[INFO] Compiling 4 source files to
/Users/tlemeur/git/surefire-junit-tests/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.21-SNAPSHOT:test (default-test) @ main ---
[WARNING] Missing POM for
org.apache.maven.surefire:surefire-booter:jar:2.21-SNAPSHOT
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.ATest
Start: ATest.methodA1[p0]
End: ATest.methodA1[p0]
Start: ATest.methodA1[p1]
End: ATest.methodA1[p1]
Start: ATest.methodA2[p0]
Start: ATest.methodA2[p1]
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.112 s
<<< FAILURE! - in com.appnexus.viewability.core.surefireJunitTests.ATest
[ERROR] methodA1[p0](com.appnexus.viewability.core.surefireJunitTests.ATest)
Time elapsed: 0.112 s <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p0]
at
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)
[ERROR] methodA1[p1](com.appnexus.viewability.core.surefireJunitTests.ATest)
Time elapsed: 0.112 s <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p1]
at
com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.CTest
Start: CTest.methodC1[p0]
End: CTest.methodC1[p0]
Start: CTest.methodC2[p0]
End: CTest.methodC2[p0]
Start: CTest.methodC1[p1]
End: CTest.methodC1[p1]
Start: CTest.methodC2[p1]
End: CTest.methodC2[p1]
Start: CTest.methodC1[p2]
End: CTest.methodC1[p2]
Start: CTest.methodC2[p2]
End: CTest.methodC2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.524 s
- in com.appnexus.viewability.core.surefireJunitTests.CTest
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.BTest
Start: BTest.methodB2[p0]
End: BTest.methodB2[p0]
Start: BTest.methodB1[p1]
End: BTest.methodB1[p1]
Start: BTest.methodB1[p0]
End: BTest.methodB1[p0]
Start: BTest.methodB2[p1]
End: BTest.methodB2[p1]
Start: BTest.methodB1[p2]
End: BTest.methodB1[p2]
Start: BTest.methodB2[p2]
End: BTest.methodB2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.312 s
- in com.appnexus.viewability.core.surefireJunitTests.BTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ATest.methodA1:14 Failing test: ATest.methodA1[p0]
[ERROR] ATest.methodA1:14 Failing test: ATest.methodA1[p1]
[INFO]
[ERROR] Tests run: 16, Failures: 2, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 4.683 s
[INFO] Finished at: 2017-06-27T15:33:49+02:00
[INFO] Final Memory: 17M/169
[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066568#comment-16066568
]
Tanguy Le Meur commented on SUREFIRE-1264:
--
[~tibor17]:
Thank you. I updated the git repo accordingly.
1) I added a class and now I think it's even harder to have the branch to not
show the problem. (99%+ of the time)
2) I'm using the same versions as you. I reproduce on Mac & Linux (I haven't
tried on Windows).
3) I tried also [~rsmith86] code, and it shown the problem too sometimes
(roughly 5% of the time).
4) That's not the problem here, but the run order seems not really
reproducible. But maybe it's only due to the logging order?
As an example
using "alphabetical", I sometimes see CBAD, CABD, CDAB... < I don't know why it
always start with C
"reversealphabetical": DCBA, CDBA, < It looks to work as I expect.
5) This order thing is not the problem, I reproduce it independentely of the
order setting. It looked to be dependant only sometimes due to a previous
version that had different test durations.
> Some tests can be lost when running in parallel with parameterized tests
>
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with
> parameterized tests. Here is a [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec -
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
> even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066618#comment-16066618
]
Tanguy Le Meur commented on SUREFIRE-1264:
--
[~tibor17]: that's interesting:
It looks like my branch work fine with surefire 2.20 (public version) but it
does not with 2.21-SNAPSHOT.
There was a problem before too for sure, but my branch is not showing it.
> Some tests can be lost when running in parallel with parameterized tests
>
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with
> parameterized tests. Here is a [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec -
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
> even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072218#comment-16072218
]
Tanguy Le Meur commented on SUREFIRE-1264:
--
Hi [~tibor17],
I just did a few changes to highlight the issue, I tried to make the issue more
reproducible.
As I understand the issue lies in the Assert.fail. If a test "fail" then no
additional test from this class are started (maybe except if they are already
in queue).
To make it more obvious, I just made all tests to Assert.fail.
I also added a class (DTest) for the same reason. So both threads (out of 2)
can trigger the issue. (Each thread slot will generally handle 2 classes).
Note also that all is key here to reproduce more easily
the issue (it also happens with the other configs).
This statements are just guesses, and are not based on actual facts as I was
not really able to understand the underlying code. This is just based on a ton
of tries (well of course with some automation :D)
As you suggest, I'll try the different versions to see the output with each
commit.
> Some tests can be lost when running in parallel with parameterized tests
>
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with
> parameterized tests. Here is a [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec -
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
> even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072573#comment-16072573
]
Tanguy Le Meur commented on SUREFIRE-1264:
--
I confirm that the "bug" with Assert.fail tests has been introduced in
5593bf560aa8d8ed900fb146176a39ad5b0e42af
Before, the number of test
That being said, before, there was already a bug. So to sum up:
* Before 5593bf560aa8d8ed900fb146176a39ad5b0e42af:
** passing tests lead to omitted tests as described in the current ticket:
SUREFIRE-1264.
** Assert.fail don't cause this problem
* After 5593bf560aa8d8ed900fb146176a39ad5b0e42af:
** Assert.fail tests lead to omitted tests.
** passing tests work fine and are well counted.
** Assume.assert have the same problem as Assert.fail tests.@Ignore test work
fine.
> Some tests can be lost when running in parallel with parameterized tests
>
>
> Key: SUREFIRE-1264
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
> Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
> Fix For: 2.20.1
>
> Attachments: failure-2.21-snapshot-SUREFIRE-1264.zip, pom.xml
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with
> parameterized tests. Here is a [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
> used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on
> the first run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [
> "$(grep -c 'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec -
> in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master
> branch|https://github.com/jderrien/surefire-junit-tests/tree/master] and it's
> even more frequent/easy to reproduce.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SUREFIRE-1263) No class name in log when running tests in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452292#comment-15452292
]
Tanguy Le Meur commented on SUREFIRE-1263:
--
Hi [~tibor17],
Using Jean-Luc repo and what you suggested, I tried to have a look:
My best guess is that the name parameter in
BlockJUnit4ClassRunnerWithParameters has been used and it's {code}name =
[p0]{code} for example (seen with the debugger). In place of a fully qualified
name.
I forked it here (look for the {code}Overriden Here{code} text):
https://github.com/tanguylemeur/surefire-junit-tests/blob/master/src/main/java/org/junit/runners/parameterized/BlockJUnit4ClassRunnerWithParameters.java
With this, the name are well reported in the log.
It's really dirty of course, but that was the closest way I found to display
the right message.
(https://cdn-images-1.medium.com/max/455/1*snTXFElFuQLSFDnvZKJ6IA.png)
I don't know who is right: junit or surefire.
I noticed that changing junit is messing up some other tools like eclipse with
this change.
> No class name in log when running tests in parallel with parameterized tests
>
>
> Key: SUREFIRE-1263
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1263
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
>
> Hello,
> Using surefire and Junit, the class names are not displayed in the output log
> when the tests are run in parallel with parameterized tests.
> According to this [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1], in
> parallel mode:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 7.799 s
> [INFO] Finished at: 2016-07-27T15:03:37+02:00
> [INFO] Final Memory: 18M/213M
> [INFO]
>
> {noformat}
> Without parallel mode (we have the 'Running #classname#' in the output):
> {noformat}
> ---
> T E S T S
> ---
> Running com.appnexus.viewability.core.surefireJunitTests.ATest
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.085 sec -
> in com.appnexus.viewability.core.surefireJunitTests.ATest
> Running com.appnexus.viewability.core.surefireJunitTests.BTest
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec -
> in com.appnexus.viewability.core.surefireJunitTests.BTest
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> It would be great to have the classname displayed when running tests in
> parallel.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (SUREFIRE-1263) No class name in log when running tests in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452292#comment-15452292
]
Tanguy Le Meur edited comment on SUREFIRE-1263 at 8/31/16 1:50 PM:
---
Hi [~tibor17],
Using Jean-Luc repo and what you suggested, I tried to have a look:
My best guess is that the name parameter in
BlockJUnit4ClassRunnerWithParameters has been used and it's {{name = [p0]}} for
example (seen with the debugger). In place of a fully qualified name.
I forked it here (look for the {{Overriden Here}} text):
https://github.com/tanguylemeur/surefire-junit-tests/blob/master/src/main/java/org/junit/runners/parameterized/BlockJUnit4ClassRunnerWithParameters.java
With this, the name are well reported in the log.
It's really dirty of course, but that was the closest way I found to display
the right message.
(https://cdn-images-1.medium.com/max/455/1*snTXFElFuQLSFDnvZKJ6IA.png)
I don't know who is right: junit or surefire.
I noticed that changing junit is messing up some other tools like eclipse with
this change.
was (Author: tlemeur):
Hi [~tibor17],
Using Jean-Luc repo and what you suggested, I tried to have a look:
My best guess is that the name parameter in
BlockJUnit4ClassRunnerWithParameters has been used and it's {code}name =
[p0]{code} for example (seen with the debugger). In place of a fully qualified
name.
I forked it here (look for the {code}Overriden Here{code} text):
https://github.com/tanguylemeur/surefire-junit-tests/blob/master/src/main/java/org/junit/runners/parameterized/BlockJUnit4ClassRunnerWithParameters.java
With this, the name are well reported in the log.
It's really dirty of course, but that was the closest way I found to display
the right message.
(https://cdn-images-1.medium.com/max/455/1*snTXFElFuQLSFDnvZKJ6IA.png)
I don't know who is right: junit or surefire.
I noticed that changing junit is messing up some other tools like eclipse with
this change.
> No class name in log when running tests in parallel with parameterized tests
>
>
> Key: SUREFIRE-1263
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1263
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
>
> Hello,
> Using surefire and Junit, the class names are not displayed in the output log
> when the tests are run in parallel with parameterized tests.
> According to this [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1], in
> parallel mode:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 7.799 s
> [INFO] Finished at: 2016-07-27T15:03:37+02:00
> [INFO] Final Memory: 18M/213M
> [INFO]
>
> {noformat}
> Without parallel mode (we have the 'Running #classname#' in the output):
> {noformat}
> ---
> T E S T S
> ---
> Running com.appnexus.viewability.core.surefireJunitTests.ATest
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJuni
[jira] [Commented] (SUREFIRE-1263) No class name in log when running tests in parallel with parameterized tests
[
https://issues.apache.org/jira/browse/SUREFIRE-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426623#comment-16426623
]
Tanguy Le Meur commented on SUREFIRE-1263:
--
Hi [~tibor17]: Yes it's totally fixed.Thanks a lot!
We haven't seen this issue since then, using various configurations.
> No class name in log when running tests in parallel with parameterized tests
>
>
> Key: SUREFIRE-1263
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1263
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.7+ (parallel) support
>Affects Versions: 2.19.1
>Reporter: Jean-Luc Derrien
>Assignee: Tibor Digana
>Priority: Major
>
> Hello,
> Using surefire and Junit, the class names are not displayed in the output log
> when the tests are run in parallel with parameterized tests.
> According to this [small
> project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1], in
> parallel mode:
> {noformat}
> ---
> T E S T S
> ---
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec -
> in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec -
> in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 7.799 s
> [INFO] Finished at: 2016-07-27T15:03:37+02:00
> [INFO] Final Memory: 18M/213M
> [INFO]
>
> {noformat}
> Without parallel mode (we have the 'Running #classname#' in the output):
> {noformat}
> ---
> T E S T S
> ---
> Running com.appnexus.viewability.core.surefireJunitTests.ATest
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 -
> sleeptime = 5 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 -
> sleeptime = 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 -
> sleeptime = 1 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.085 sec -
> in com.appnexus.viewability.core.surefireJunitTests.ATest
> Running com.appnexus.viewability.core.surefireJunitTests.BTest
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec -
> in com.appnexus.viewability.core.surefireJunitTests.BTest
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> It would be great to have the classname displayed when running tests in
> parallel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
