[
https://issues.apache.org/jira/browse/SUREFIRE-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cam Morris updated SUREFIRE-1228:
---------------------------------
Attachment: TestRandomFail.java
This file is placed in src/test/java directory
> rerunFailingTestsCount + TestNG + @DataProvider = failed tests pass
> -------------------------------------------------------------------
>
> Key: SUREFIRE-1228
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1228
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
> Affects Versions: 2.19.1
> Reporter: Cam Morris
> Attachments: TestRandomFail.java
>
>
> Surefire confuses tests run with a data-provider with rerun tests so only one
> of the data-provided tests need to pass.
> Steps to reproduce:
> # Create a testNG test with a data provider that passes with some of the data
> and fails with others
> {code:title=TestNG example|language=java}
> @Test(dataProvider = "succeed")
> public void fail(boolean succeed) {
> if (!succeed) {
> Assert.fail("nope");
> }
> }
> @DataProvider(name="succeed")
> public Object[][] arrayIndex() {
> return new Object[][]{ {true}, {false}};
> }
> {code}
> # Configure surefire to rerun failed tests
> {code:title=pom.xml example|language=xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> <modelVersion>4.0.0</modelVersion>
> <groupId>cam.test</groupId>
> <artifactId>testng-retry</artifactId>
> <version>1.0-SNAPSHOT</version>
> <dependencies>
> <dependency>
> <groupId>org.testng</groupId>
> <artifactId>testng</artifactId>
> <version>6.9.9</version>
> <scope>test</scope>
> </dependency>
> </dependencies>
> <build>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.19.1</version>
> <configuration>
> <rerunFailingTestsCount>1</rerunFailingTestsCount>
> </configuration>
> </plugin>
> </plugins>
> </build>
> </project>
> {code}
> - *Expected results*: "tests run: 2, Failures: 1..." The tests should not
> pass.
> - *Actual results*: The different runs using the provider data are confused
> for retry attempts. Just one of the data needs to succeed for the tests to
> be assumed a success.
> {code}mvn test
> ...
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running TestRandomFail
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec <<<
> FAILURE! - in TestRandomFail
> fail(TestRandomFail) Time elapsed: 0.007 sec <<< FAILURE!
> java.lang.AssertionError: nope
> at TestRandomFail.fail(TestRandomFail.java:14)
> Results :
> Flaked tests:
> TestRandomFail.fail(TestRandomFail)
> Run 1: PASS
> Run 2: TestRandomFail.fail:14 nope
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 2.251 s
> [INFO] Finished at: 2016-02-10T07:39:21-07:00
> [INFO] Final Memory: 9M/244M
> [INFO]
> ------------------------------------------------------------------------
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
