[
https://issues.apache.org/jira/browse/SUREFIRE-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Skarzhevskyy updated SUREFIRE-1760:
----------------------------------------
Description:
Running JUnit5 tests using maven-surefire-plugin with
<reuseForks>true</reuseForks> e.g. mvn -DforkCount=8
Results: number of tests in each fork is significantly unbalanced, *+some forks
do not get tests at all+*..
The end results is that on CI server the total test execution will take as long
as longest fork and only 50% of the available resources are used.
In our case we observed that our 3000 integration tests after migration to
junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.
The balance is restored when running with reuseForks=false but this will cause
additional overhead for complex integrations tests
e.g. tests requiring additional setup , classpath scanning and so on..
The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4
Maven 3.6.3 , openjdk version "1.8.0_242" on Linux
BTW If junit-vintage-engine is used the same results observed for JUnit3 and
JUnit4 tests.
Consider example of 100 tests (10 classes 10 tests in each)
[https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork]
When running mvn -DforkCount=8 with extension that prints the number of tests
executed in each fork getting following:
{quote}Tests completed JVM Fork 8; total tests: 0; classes: 0
Tests completed JVM Fork 4; total tests: 0; classes: 0
Tests completed JVM Fork 1; total tests: 0; classes: 0
Tests completed JVM Fork 6; total tests: 0; classes: 0
Tests completed JVM Fork 3; total tests: 0; classes: 0
Tests completed JVM Fork 2; total tests: 0; classes: 0
Tests completed JVM Fork 7; total tests: 30; classes: 3
Tests completed JVM Fork 5; total tests: 70; classes: 7
{quote}
mvn -DforkCount=4
{quote}Tests completed JVM Fork 2; total tests: 0; classes: 0
Tests completed JVM Fork 3; total tests: 10; classes: 1
Tests completed JVM Fork 4; total tests: 40; classes: 4
Tests completed JVM Fork 1; total tests: 50; classes: 5
{quote}
The distribution is also random, results may vary between maven runs:
mvn -DforkCount=4
{quote}Tests completed JVM Fork 4; total tests: 0; classes: 0
Tests completed JVM Fork 2; total tests: 10; classes: 1
Tests completed JVM Fork 1; total tests: 30; classes: 3
Tests completed JVM Fork 3; total tests: 60; classes: 6
{quote}
was:
Running JUnit5 tests using maven-surefire-plugin with
<reuseForks>true</reuseForks> e.g. mvn -DforkCount=8
Results: number of tests in each fork is significantly unbalanced, *+some forks
do not get tests at all+*..
The end results is that on CI server the total test execution will take as long
as longest fork and only 50% of the available resources are used.
In our case we observed that our 3000 integration tests after migration to
junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.
The balance is restored when running with reuseForks=false but this will cause
additional overhead for complex integrations tests
e.g. tests requiring additional setup , classpath scanning and so on..
The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4
Maven 3.6.3 , openjdk version "1.8.0_242" on Linux
BTW If junit-vintage-engine is used the same results observed for JUnit3 and
JUnit4 tests.
Consider example of 100 tests (10 classes 10 tests in each)
[https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork]
When running mvn -DforkCount=8 with extension that prints the number of tests
executed in each fork getting following:
{quote}Tests completed JVM Fork 8; total tests: 0; classes: 0
Tests completed JVM Fork 4; total tests: 0; classes: 0
Tests completed JVM Fork 1; total tests: 0; classes: 0
Tests completed JVM Fork 6; total tests: 0; classes: 0
Tests completed JVM Fork 3; total tests: 0; classes: 0
Tests completed JVM Fork 2; total tests: 0; classes: 0
Tests completed JVM Fork 7; total tests: 30; classes: 3
Tests completed JVM Fork 5; total tests: 70; classes: 7{quote}
mvn -DforkCount=4
{quote}Tests completed JVM Fork 2; total tests: 0; classes: 0
Tests completed JVM Fork 3; total tests: 10; classes: 1
Tests completed JVM Fork 4; total tests: 40; classes: 4
Tests completed JVM Fork 1; total tests: 50; classes: 5{quote}
The distribution is also random, results may vary between maven runs:
mvn -DforkCount=4
{quote}Tests completed JVM Fork 4; total tests: 0; classes: 0
Tests completed JVM Fork 2; total tests: 10; classes: 1
Tests completed JVM Fork 1; total tests: 30; classes: 3
Tests completed JVM Fork 3; total tests: 60; classes: 6{quote}
> Unbalanced number of test executions in forks (forkCount>1) results in no
> performance gain for jupiter/JUnit5 tests
> -------------------------------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1760
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1760
> Project: Maven Surefire
> Issue Type: Bug
> Components: JUnit 5.x support
> Reporter: Vlad Skarzhevskyy
> Priority: Major
>
> Running JUnit5 tests using maven-surefire-plugin with
> <reuseForks>true</reuseForks> e.g. mvn -DforkCount=8
> Results: number of tests in each fork is significantly unbalanced, *+some
> forks do not get tests at all+*..
> The end results is that on CI server the total test execution will take as
> long as longest fork and only 50% of the available resources are used.
> In our case we observed that our 3000 integration tests after migration to
> junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.
> The balance is restored when running with reuseForks=false but this will
> cause additional overhead for complex integrations tests
> e.g. tests requiring additional setup , classpath scanning and so on..
> The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4
> Maven 3.6.3 , openjdk version "1.8.0_242" on Linux
>
> BTW If junit-vintage-engine is used the same results observed for JUnit3 and
> JUnit4 tests.
>
> Consider example of 100 tests (10 classes 10 tests in each)
>
> [https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork]
>
> When running mvn -DforkCount=8 with extension that prints the number of
> tests executed in each fork getting following:
> {quote}Tests completed JVM Fork 8; total tests: 0; classes: 0
> Tests completed JVM Fork 4; total tests: 0; classes: 0
> Tests completed JVM Fork 1; total tests: 0; classes: 0
> Tests completed JVM Fork 6; total tests: 0; classes: 0
> Tests completed JVM Fork 3; total tests: 0; classes: 0
> Tests completed JVM Fork 2; total tests: 0; classes: 0
> Tests completed JVM Fork 7; total tests: 30; classes: 3
> Tests completed JVM Fork 5; total tests: 70; classes: 7
> {quote}
>
> mvn -DforkCount=4
> {quote}Tests completed JVM Fork 2; total tests: 0; classes: 0
> Tests completed JVM Fork 3; total tests: 10; classes: 1
> Tests completed JVM Fork 4; total tests: 40; classes: 4
> Tests completed JVM Fork 1; total tests: 50; classes: 5
> {quote}
>
> The distribution is also random, results may vary between maven runs:
> mvn -DforkCount=4
> {quote}Tests completed JVM Fork 4; total tests: 0; classes: 0
> Tests completed JVM Fork 2; total tests: 10; classes: 1
> Tests completed JVM Fork 1; total tests: 30; classes: 3
> Tests completed JVM Fork 3; total tests: 60; classes: 6
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
