Falko Modler created SUREFIRE-1169:
--------------------------------------
Summary: JUnit / Arquillian lifecycle friendly test execution with
forkCount > 1 and reusableForks = true
Key: SUREFIRE-1169
URL: https://issues.apache.org/jira/browse/SUREFIRE-1169
Project: Maven Surefire
Issue Type: Improvement
Components: process forking
Affects Versions: 2.18.1
Reporter: Falko Modler
The current approach to "stream" each test class name to a fork via
{{TestProvidingInputStream}} yields a good "load balancing" accross the forks
but it *triggers the entire test lifecycle for each test*.
With {{forkCount = 1}}, all tests are executed in one set but with {{forkCount
= n}} (n > 1) each test is a separate "set" (so to say).
This is very problematic in case you or a test framwork you are using relies on
a certain lifecycle.
[Arquillian|http://arquillian.org/] for example ties various "events" to
JUnit's lifecycle methods, like {{AfterSuite}} to
{{RunListener.testRunFinished(...)}} which triggers the shutdown of the managed
server etc.
When using multiple forks, {{RunListener.testRunFinished(...)}} is called for
*every* single test class, triggering {{AfterSuite}} for every single test,
although the fork will receive further tests after that which should "reuse"
the server.
This is just an example. In fact the whole JUnit / Arquillian lifecycle is
inconsistent when using multiple forks.
>From a user perspective I wouldn't expect this behaviour:
As {{forkCount = 1}} (and {{reusableForks = true}}) triggers
{{RunListener.testRunFinished(...)}} *once*, {{forkCount = n}} (and
{{reusableForks = true}}) should trigger that method *n* times, not *x* times.
To be fair, the
[documentation|https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html]
*does* contain a pointer to that problem by saying:
{quote}
When using reuseForks=true and a forkCount value larger than one, test classes
are handed over to the forked process one-by-one.
{quote}
But the consequences remain very unclear.
*(Possible) Solution:*
I took a stab at this and implemented an "eager test distribution" to the forks
in {{ForkStart.java}} and disabled streaming. Please see attached patch (to be
applied against project root, 2.18.1).
Patch Details:
- New config property: {{ForkConfiguration.forksExecuteTestsOneByOne}}, set via
Mojo (default is true for backward compatibility, name is debatable)
- When {{forksExecuteTestsOneByOne}} ist set to false, the {{messageQueue}} in
{{ForkStarter.runSuitesForkOnceMultiple(...)}} is *not* wrapped in fork
specific {{TestProvidingInputStream}} instances to be consumed bit by bit later
on.
Instead, the queue is consumed directly and each test class name is assigned to
the respective fork by creating a copy of the {{providerProperties}} which is
filled individually for each fork.
E.g. for three forks and eight tests:
-- fork 1 executes test 1, 4 and 7
-- fork 2 executes test 2, 5 and 8
-- fork 3 executes test 3 and 6
- To have a clean {{providerProperties}} template I had to move
{{DefaultScanResult.writeTo(...)}} to the respective private methods. Otherwise
the properties would have contained *all* tests already.
- I refactored some methods in {{ForkStarter}} to enhance readability and to
reduce code duplication.
- The patch does *not* contain a test for the new behaviour but all existing
tests passed.
{{forksExecuteTestsOneByOne = false}} now leads to a consistent lifecycle.
This solution has one downside: One or more forks could be overloaded while
other forks could "underloaded" because you cannot say how long each test will
take. runOrder=balanced could help here but has yet to be implemented for forks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
