[
https://jira.codehaus.org/browse/SUREFIRE-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=355928#comment-355928
]
Benoit Lagae commented on SUREFIRE-1113:
----------------------------------------
Nice catch ! I hadn't tried different versions of TestNG, never really thought
about it because I've always used 6.8+
I can confirm that the code gives the expected results until TestNG 5.13.1, so
the point where this bug has been introduced seems to be TestNG 5.14. I will
log a bug with TestNG, referring to this issue.
> Build does not fail when successPercentage for @org.testng.annotations.Test()
> is not met
> ----------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1113
> URL: https://jira.codehaus.org/browse/SUREFIRE-1113
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin, TestNG support
> Affects Versions: 2.18
> Environment: TestNG 6.8.8, maven-surefire-plugin 2.19-SNAPSHOT
> (53a40eef48ea)
> Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4;
> 2014-08-11T22:58:10+02:00)
> Maven home: C:\Program Files\apache-maven-3.2.3
> Java version: 1.7.0_67, vendor: Oracle Corporation
> Java home: C:\Program Files\Java\jdk1.7.0_67\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
> Reporter: Benoit Lagae
> Assignee: Andreas Gudian
> Attachments: surefiretest.zip
>
>
> When a TestNG test with annotation methods invocationCount = x and
> successPercentage = y fails, this does not break the Maven build of the
> project. I have attached a minimal Maven project with only the pom and a
> single test class, which contains three dummy tests which should all fail:
> {code}
> @Test
> public void testFailsCorrectly() { fail("dummy"); }
> @Test(invocationCount = x)
> public void testCounterWorksAndTestFails() { fail("dummy"); }
> @Test(invocationCount = x, successPercentage = y)
> public void testShouldFailButDoesnt() { fail("dummy"); }
> {code}
> The first two comply, but the third one doesn't, as is implied in the name of
> the method. I created this project from scratch, i.e. not from an IDE: manual
> creation of folders, code and pom in Notepad++, ...
> I have attached both the project and the info I thought could be relevant
> from my test runs of the project ('diagnostics' folder). Below are the
> results of the individual method runs:
> {code}
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testFailsCorrectly()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testFailsCorrectly"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE"
> signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE"
> signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> {code}
> If we comment out the other tests, and only make testShouldFailButDoesnt run,
> then the build succeeds while it shouldn't. See the results in
> surefiretest/diagnostics/only_faulty_test/console_output.txt
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)
