[ https://issues.apache.org/jira/browse/SUREFIRE-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631416#comment-14631416 ]
Tibor Digana edited comment on SUREFIRE-1169 at 7/17/15 2:40 PM: ----------------------------------------------------------------- [~famod] Maybe not necessary to make too big changes. You can see JUnit4Provicer and JUnitCoreProvider and wrapper. AFAIK you want the surefire plugin to call testRunStarted and testRunFinished just once per JVM. If Arquillian specifies system property, -Dlistener=JUnitListener, then it is just a question to suppress multiple calls on custom listeners. Right? [~agudian] Can we enable just only one call of testRunStarted and testRunFinished on custom listeners per JVM? Would it have any negative side effect on surefire functionality? was (Author: tibor17): [~famod] Maybe not necessary to make too big changes. You can see JUnit4Provicer and JUnitCoreProvider and wrapper. AFAIK all you want the surefire plugin to call testRunStarted and testRunFinished called just once per JVM. If Arquillian specifies system property, -Dlistener=JUnitListener, then it is just a question to suppress multiple calls on custom listeners. Right? [~agudian] Can we enable just only one call of testRunStarted and testRunFinished on custom listeners per JVM? Would it have any negative side effect on surefire functionality? > JUnit / Arquillian lifecycle friendly test execution with forkCount > 1 and > reuseForks = true > --------------------------------------------------------------------------------------------- > > Key: SUREFIRE-1169 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1169 > Project: Maven Surefire > Issue Type: Improvement > Components: process forking > Affects Versions: 2.18.1 > Reporter: Falko Modler > Attachments: SUREFIRE-1169_forksExecuteTestsOneByOne.patch > > > The current approach to "stream" each test class name to a fork via > {{TestProvidingInputStream}} yields a good "load balancing" accross the forks > but it *triggers the entire test lifecycle for each test*. > With {{forkCount = 1}}, all tests are executed in one set but with > {{forkCount = n}} (n > 1) each test is a separate "set" (so to say). > This is very problematic in case you or a test framwork you are using relies > on a certain lifecycle. > [Arquillian|http://arquillian.org/] for example ties various "events" to > JUnit's lifecycle methods, like {{AfterSuite}} to > {{RunListener.testRunFinished(...)}} which triggers the shutdown of the > managed server etc. > When using multiple forks, {{RunListener.testRunFinished(...)}} is called for > *every* single test class, triggering {{AfterSuite}} for every single test, > although the fork will receive further tests after that which should "reuse" > the server. > This is just an example. In fact the whole JUnit / Arquillian lifecycle is > inconsistent when using multiple forks. > From a user perspective I wouldn't expect this behaviour: > As {{forkCount = 1}} (and {{reusableForks = true}}) triggers > {{RunListener.testRunFinished(...)}} *once*, {{forkCount = n}} (and > {{reusableForks = true}}) should trigger that method *n* times, not *x* times. > To be fair, the > [documentation|https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html] > *does* contain a pointer to that problem by saying: > {quote} > When using reuseForks=true and a forkCount value larger than one, test > classes are handed over to the forked process one-by-one. > {quote} > But the consequences remain very unclear. > *(Possible) Solution:* > I took a stab at this and implemented an "eager test distribution" to the > forks in {{ForkStart.java}} and disabled streaming. Please see attached patch > (to be applied against project root, 2.18.1). > Patch Details: > - New config property: {{ForkConfiguration.forksExecuteTestsOneByOne}}, set > via Mojo (default is true for backward compatibility, name is debatable) > - When {{forksExecuteTestsOneByOne}} ist set to false, the {{messageQueue}} > in {{ForkStarter.runSuitesForkOnceMultiple(...)}} is *not* wrapped in fork > specific {{TestProvidingInputStream}} instances to be consumed bit by bit > later on. > Instead, the queue is consumed directly and each test class name is assigned > to the respective fork by creating a copy of the {{providerProperties}} which > is filled individually for each fork. > E.g. for three forks and eight tests: > -- fork 1 executes test 1, 4 and 7 > -- fork 2 executes test 2, 5 and 8 > -- fork 3 executes test 3 and 6 > - To have a clean {{providerProperties}} template I had to move > {{DefaultScanResult.writeTo(...)}} to the respective private methods. > Otherwise the properties would have contained *all* tests already. > - I refactored some methods in {{ForkStarter}} to enhance readability and to > reduce code duplication. > - The patch does *not* contain a test for the new behaviour but all existing > tests passed. > {{forksExecuteTestsOneByOne = false}} now leads to a consistent lifecycle. > This solution has one downside: One or more forks could be overloaded while > other forks could "underloaded" because you cannot say how long each test > will take. runOrder=balanced could help here but has yet to be implemented > for forks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
