[
https://jira.codehaus.org/browse/SUREFIRE-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=358639#comment-358639
]
Roy Arnon edited comment on SUREFIRE-1122 at 12/4/14 4:12 PM:
--------------------------------------------------------------
In my pom file, I have a profile called parallel that defines some options for
failsafe to run parallely:
{code:xml}
<profile>
<id>parallel</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<reuseForks>false</reuseForks>
<threadCountSuites>2</threadCountSuites>
<threadCountClasses>2</threadCountClasses>
<threadCountMethods>6</threadCountMethods>
<forkCount>${parallel.forkcount}</forkCount>
<threadCount>${parallel.threadcount}</threadCount>
<parallel>all</parallel>
</configuration>
</plugin>
</plugins>
</build>
</profile>
{code}
When I run the tests without this profile, I get a flakyFailure tag in the
testcase as defined
[here|http://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html]:
{code:xml}
<testcase name="loginWithGoodUserDontRememberThenLogout"
classname="com.mycompany.testing.backstage.general.login.LoginTest"
time="6.297">
<flakyFailure message="Expected to be able to login, but got back to login
page Expected: an instance of
com.mycompany.testing.pages.backstage.MainPage but:
<com.mycompany.testing.pages.backstage.login.LoginPage@1df95673> is a
com.mycompany.testing.pages.backstage.login.LoginPage"
type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError: Expected to
be able to login, but got back to login page
Expected: an instance of com.mycompany.testing.pages.backstage.MainPage
but: <com.mycompany.testing.pages.backstage.login.LoginPage@1df95673> is a
com.mycompany.testing.pages.backstage.login.LoginPage
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:865)
at
com.mycompany.testing.backstage.general.login.LoginTest.loginWithGoodUserDontRememberThenLogout(LoginTest.java:60)
]]>
<system-out><![CDATA[17:08:08.011 [INFO] Created dir: target/screenshots/
[[ATTACHMENT|c:\workspace\products\automation-testing\backstage-test-suite\target\logs\LoginTest.loginWithGoodUserDontRememberThenLogout.log]]
Unable to obtain hostname of webdriver node, assuming localhost
17:08:08.029 [INFO] Selenium Node: ROY-DESKTOP.office.mycompany.com
17:08:09.675 [INFO] Logging in user: login_user
17:08:09.810 [INFO] Entering text [login_user] to User Name located by
[[[ChromeDriver: chrome on XP (45adbb70264e3dde486c0fc57eadbb1f)] -> id:
j_username]]
17:08:10.049 [INFO] Entering text [change121] to Password located by
[[[ChromeDriver: chrome on XP (45adbb70264e3dde486c0fc57eadbb1f)] -> id:
j_password]]
17:08:10.365 [WARN] Got empty URL path, validating on page using sign out button
17:08:10.533 [ERROR] Failed logging in user: login_user. Got error: [The
username or password you entered is incorrect]
17:08:10.540 [INFO] Test Failed
17:08:10.543 [INFO] Current URL:
http://qa-fe-auto4-backstage.mycompany.com/backstage/login#
17:08:10.549 [ERROR] Error in test:
LoginTest_loginWithGoodUserDontRememberThenLogout, Taking screenshot
Current URL: http://qa-fe-auto4-backstage.mycompany.com/backstage/login#
17:08:10.549 [INFO] Taking screenshot with name:
LoginTest_loginWithGoodUserDontRememberThenLogout
[[ATTACHMENT|c:\workspace\products\automation-testing\backstage-test-suite\target\screenshots\LoginTest_loginWithGoodUserDontRememberThenLogout.png]]
]]></system-out>
<system-err><![CDATA[Starting ChromeDriver 2.12.301325
(962dea43ddd90e7e4224a03fa3c36a421281abb7) on port 14466
Only local connections are allowed.
]]></system-err>
</flakyFailure>
</testcase>
{code}
When I run the tests with the parallel option, this is the testcase tag:
{code:xml}
<testcase name="loginWithGoodUserDontRememberThenLogout"
classname="com.mycompany.testing.backstage.general.login.LoginTest"
time="9.744"/>
{code}
The test is a simple selenium test that has been changed by me to only succeed
on the 2nd run.
I've also attached a simple project with a single test, that you can reproduce
the bug on - simply run once with the -Pparallel option, and once without -
check out the the resulting xml file.
There is also another issue when running with the parallel profile - the
number of tests in the test output xml file does not match the number of tests
actually run.
If you try running the test program with the -Pparallel option, you should
notice only one testcase in the resulting output xml file, and not 2 as it
should have been.
If you require anything else, please let me know.
was (Author: hellspam):
In my pom file, I have a profile called parallel that defines some options for
failsafe to run parallely:
{code:xml}
<profile>
<id>parallel</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<reuseForks>false</reuseForks>
<threadCountSuites>2</threadCountSuites>
<threadCountClasses>2</threadCountClasses>
<threadCountMethods>6</threadCountMethods>
<forkCount>${parallel.forkcount}</forkCount>
<threadCount>${parallel.threadcount}</threadCount>
<parallel>all</parallel>
</configuration>
</plugin>
</plugins>
</build>
</profile>
{code}
When I run the tests without this profile, I get a flakyFailure tag in the
testcase as defined
[here|http://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html]:
{code:xml}
<testcase name="loginWithGoodUserDontRememberThenLogout"
classname="com.mycompany.testing.backstage.general.login.LoginTest"
time="6.297">
<flakyFailure message="Expected to be able to login, but got back to login
page Expected: an instance of
com.mycompany.testing.pages.backstage.MainPage but:
<com.mycompany.testing.pages.backstage.login.LoginPage@1df95673> is a
com.mycompany.testing.pages.backstage.login.LoginPage"
type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError: Expected to
be able to login, but got back to login page
Expected: an instance of com.mycompany.testing.pages.backstage.MainPage
but: <com.mycompany.testing.pages.backstage.login.LoginPage@1df95673> is a
com.mycompany.testing.pages.backstage.login.LoginPage
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:865)
at
com.mycompany.testing.backstage.general.login.LoginTest.loginWithGoodUserDontRememberThenLogout(LoginTest.java:60)
]]>
<system-out><![CDATA[17:08:08.011 [INFO] Created dir: target/screenshots/
[[ATTACHMENT|c:\workspace\products\automation-testing\backstage-test-suite\target\logs\LoginTest.loginWithGoodUserDontRememberThenLogout.log]]
Unable to obtain hostname of webdriver node, assuming localhost
17:08:08.029 [INFO] Selenium Node: ROY-DESKTOP.office.mycompany.com
17:08:09.675 [INFO] Logging in user: login_user
17:08:09.810 [INFO] Entering text [login_user] to User Name located by
[[[ChromeDriver: chrome on XP (45adbb70264e3dde486c0fc57eadbb1f)] -> id:
j_username]]
17:08:10.049 [INFO] Entering text [change121] to Password located by
[[[ChromeDriver: chrome on XP (45adbb70264e3dde486c0fc57eadbb1f)] -> id:
j_password]]
17:08:10.365 [WARN] Got empty URL path, validating on page using sign out button
17:08:10.533 [ERROR] Failed logging in user: login_user. Got error: [The
username or password you entered is incorrect]
17:08:10.540 [INFO] Test Failed
17:08:10.543 [INFO] Current URL:
http://qa-fe-auto4-backstage.mycompany.com/backstage/login#
17:08:10.549 [ERROR] Error in test:
LoginTest_loginWithGoodUserDontRememberThenLogout, Taking screenshot
Current URL: http://qa-fe-auto4-backstage.mycompany.com/backstage/login#
17:08:10.549 [INFO] Taking screenshot with name:
LoginTest_loginWithGoodUserDontRememberThenLogout
[[ATTACHMENT|c:\workspace\products\automation-testing\backstage-test-suite\target\screenshots\LoginTest_loginWithGoodUserDontRememberThenLogout.png]]
]]></system-out>
<system-err><![CDATA[Starting ChromeDriver 2.12.301325
(962dea43ddd90e7e4224a03fa3c36a421281abb7) on port 14466
Only local connections are allowed.
]]></system-err>
</flakyFailure>
</testcase>
{code}
When I run the tests with the parallel option, this is the testcase tag:
{code:xml}
<testcase name="loginWithGoodUserDontRememberThenLogout"
classname="com.mycompany.testing.backstage.general.login.LoginTest"
time="9.744"/>
{code}
The test is a simple selenium test that has been changed by me to only succeed
on the 2nd run.
I've also attached a simple project with a single test, that you can reproduce
the bug on - simply run once with the -Pparallel option, and once without -
check out the the resulting xml file.
There is also another issue when running with the parallel profile - the
number of tests in the test output xml file does not match the number of
changes actually run.
If you try running the test program with the -Pparallel option, you should
notice only one testcase in the resulting output xml file, and not 2 as it
should have been.
If you require anything else, please let me know.
> When running failsafe with -Dfailsafe.rerunFailingTestsCount and parallel
> all, the reports are not generated correctly
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1122
> URL: https://jira.codehaus.org/browse/SUREFIRE-1122
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Failsafe Plugin, Maven Surefire Plugin
> Affects Versions: 2.18
> Reporter: Roy Arnon
> Attachments: invalid.xml, test.zip, valid.xml
>
>
> Hi,
> I just tried the new feature -Dfailsafe.rerunFailingTestsCount on our
> failsafe build.
> Unfortunately, when running the build using parallel=all, it seems the junit
> xml files are not generated correctly.
> I've tried running it multiple different ways, including classedAndMethods,
> and all, but it seems to generate an incorrect xml file either way.
> I've attached both files here.
> I can provide a pom and java test case if required.
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)
