[
https://issues.apache.org/jira/browse/SUREFIRE-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632506#comment-14632506
]
Falko Modler edited comment on SUREFIRE-1169 at 7/19/15 11:57 AM:
------------------------------------------------------------------
Changing the Arquillian runner seems like fixing the problem at the wrong end.
When shall the runner eventually decide that it can shutdown the server and do
all the other cleanup that is tied to the {{AfterSuite}} event?
The natural junit approach is {{testRunFinished}} and that's how it is today.
The only alternative I can spontaneously think of is a shutdown hook.
-Keeping the server up and running could lead to other problems in a mixed set
of test with some Arquillian-Tests and some non-Arquillian tests (we have that
in our project):-
-The runner would keep the server running although none of the following tests
is an Arquillian test. The server might block certain resources like ports or
databases that subsequent tests need to use exclusively.-
Edit: I was wrong about that. This is the default behaviour of the Arquillian
runner, even when using a single fork.
Other frameworks that are built on top of Junit and which are relying on
{{testRunFinished}} (and started) will potentially have the same problems like
Arquillian.
I don't think it is feasible to "fix"/change them all.
What we probably need here to have a consistent solution is an extended JUnit
API so that you can tell {{JUnitCore}} to execute a {{Request}} which is part
of a bigger "test run".
I don't see that comming (soon) though.
Until then I'd say: Please give the surefire users a possibility to execute
(Arquillians) tests in multiple reusable forks without tearing the test set/run
apart.
was (Author: famod):
Changing the Arquillian runner seems like fixing the problem at the wrong end.
When shall the runner eventually decide that it can shutdown the server and do
all the other cleanup that is tied to the {{AfterSuite}} event?
The natural junit approach is {{testRunFinished}} and that's how it is today.
The only alternative I can spontaneously think of is a shutdown hook.
Keeping the server up and running could lead to other problems in a mixed set
of test with some Arquillian-Tests and some non-Arquillian tests (we have that
in our project):
The runner would keep the server running although none of the following tests
is an Arquillian test. The server might block certain resources like ports or
databases that subsequent tests need to use exclusively.
Other frameworks that are built on top of Junit and which are relying on
{{testRunFinished}} (and started) will potentially have the same problems like
Arquillian.
I don't think it is feasible to "fix"/change them all.
What we probably need here to have a consistent solution is an extended JUnit
API so that you can tell {{JUnitCore}} to execute a {{Request}} which is part
of a bigger "test run".
I don't see that comming (soon) though.
Until then I'd say: Please give the surefire users a possibility to execute
(Arquillians) tests in multiple reusable forks without tearing the test set/run
apart.
> JUnit / Arquillian lifecycle friendly test execution with forkCount > 1 and
> reuseForks = true
> ---------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1169
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1169
> Project: Maven Surefire
> Issue Type: Improvement
> Components: process forking
> Affects Versions: 2.18.1
> Reporter: Falko Modler
> Attachments: SUREFIRE-1169_forksExecuteTestsOneByOne.patch
>
>
> The current approach to "stream" each test class name to a fork via
> {{TestProvidingInputStream}} yields a good "load balancing" accross the forks
> but it *triggers the entire test lifecycle for each test*.
> With {{forkCount = 1}}, all tests are executed in one set but with
> {{forkCount = n}} (n > 1) each test is a separate "set" (so to say).
> This is very problematic in case you or a test framwork you are using relies
> on a certain lifecycle.
> [Arquillian|http://arquillian.org/] for example ties various "events" to
> JUnit's lifecycle methods, like {{AfterSuite}} to
> {{RunListener.testRunFinished(...)}} which triggers the shutdown of the
> managed server etc.
> When using multiple forks, {{RunListener.testRunFinished(...)}} is called for
> *every* single test class, triggering {{AfterSuite}} for every single test,
> although the fork will receive further tests after that which should "reuse"
> the server.
> This is just an example. In fact the whole JUnit / Arquillian lifecycle is
> inconsistent when using multiple forks.
> From a user perspective I wouldn't expect this behaviour:
> As {{forkCount = 1}} (and {{reusableForks = true}}) triggers
> {{RunListener.testRunFinished(...)}} *once*, {{forkCount = n}} (and
> {{reusableForks = true}}) should trigger that method *n* times, not *x* times.
> To be fair, the
> [documentation|https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html]
> *does* contain a pointer to that problem by saying:
> {quote}
> When using reuseForks=true and a forkCount value larger than one, test
> classes are handed over to the forked process one-by-one.
> {quote}
> But the consequences remain very unclear.
> *(Possible) Solution:*
> I took a stab at this and implemented an "eager test distribution" to the
> forks in {{ForkStart.java}} and disabled streaming. Please see attached patch
> (to be applied against project root, 2.18.1).
> Patch Details:
> - New config property: {{ForkConfiguration.forksExecuteTestsOneByOne}}, set
> via Mojo (default is true for backward compatibility, name is debatable)
> - When {{forksExecuteTestsOneByOne}} ist set to false, the {{messageQueue}}
> in {{ForkStarter.runSuitesForkOnceMultiple(...)}} is *not* wrapped in fork
> specific {{TestProvidingInputStream}} instances to be consumed bit by bit
> later on.
> Instead, the queue is consumed directly and each test class name is assigned
> to the respective fork by creating a copy of the {{providerProperties}} which
> is filled individually for each fork.
> E.g. for three forks and eight tests:
> -- fork 1 executes test 1, 4 and 7
> -- fork 2 executes test 2, 5 and 8
> -- fork 3 executes test 3 and 6
> - To have a clean {{providerProperties}} template I had to move
> {{DefaultScanResult.writeTo(...)}} to the respective private methods.
> Otherwise the properties would have contained *all* tests already.
> - I refactored some methods in {{ForkStarter}} to enhance readability and to
> reduce code duplication.
> - The patch does *not* contain a test for the new behaviour but all existing
> tests passed.
> {{forksExecuteTestsOneByOne = false}} now leads to a consistent lifecycle.
> This solution has one downside: One or more forks could be overloaded while
> other forks could "underloaded" because you cannot say how long each test
> will take. runOrder=balanced could help here but has yet to be implemented
> for forks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
