[
https://jira.codehaus.org/browse/SUREFIRE-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=355900#comment-355900
]
Andreas Gudian commented on SUREFIRE-1113:
------------------------------------------
OK, great: the examples in SUREFIRE-654 and also the created integration test
use TestNG 5.7, and there it works. For TestNG 6.x, it's not working anymore.
What we do in Surefire is to implement {{org.testng.ITestListener}} to be
informed about started tests and the result of those tests. Fortunately, the
following method from that interface has a pretty clear documentation:
{code}
/**
* Invoked each time a method fails but has been annotated with
* successPercentage and this failure still keeps it within the
* success percentage requested.
*
* @param result <code>ITestResult</code> containing information about the
run test
* @see ITestResult#SUCCESS_PERCENTAGE_FAILURE
*/
public void onTestFailedButWithinSuccessPercentage(ITestResult result);
{code}
That method is called all four times for the test {{testShouldFailButDoesnt}}
in version 6.8.8 that you use in your pom. In TestNG 5.7, it is called only the
first two times, where the failure percentage is still under 70 percent. Given
the javadoc, I'd say that 5.7 did it right and 6.x introduced a bug here.
Could you please check back with the TestNG folks and file a bug there, if
necessary?
> Build does not fail when successPercentage for @org.testng.annotations.Test()
> is not met
> ----------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1113
> URL: https://jira.codehaus.org/browse/SUREFIRE-1113
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin, TestNG support
> Affects Versions: 2.18
> Environment: TestNG 6.8.8, maven-surefire-plugin 2.19-SNAPSHOT
> (53a40eef48ea)
> Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4;
> 2014-08-11T22:58:10+02:00)
> Maven home: C:\Program Files\apache-maven-3.2.3
> Java version: 1.7.0_67, vendor: Oracle Corporation
> Java home: C:\Program Files\Java\jdk1.7.0_67\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
> Reporter: Benoit Lagae
> Assignee: Andreas Gudian
> Attachments: surefiretest.zip
>
>
> When a TestNG test with annotation methods invocationCount = x and
> successPercentage = y fails, this does not break the Maven build of the
> project. I have attached a minimal Maven project with only the pom and a
> single test class, which contains three dummy tests which should all fail:
> {code}
> @Test
> public void testFailsCorrectly() { fail("dummy"); }
> @Test(invocationCount = x)
> public void testCounterWorksAndTestFails() { fail("dummy"); }
> @Test(invocationCount = x, successPercentage = y)
> public void testShouldFailButDoesnt() { fail("dummy"); }
> {code}
> The first two comply, but the third one doesn't, as is implied in the name of
> the method. I created this project from scratch, i.e. not from an IDE: manual
> creation of folders, code and pom in Notepad++, ...
> I have attached both the project and the info I thought could be relevant
> from my test runs of the project ('diagnostics' folder). Below are the
> results of the individual method runs:
> {code}
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testFailsCorrectly()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testFailsCorrectly"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE"
> signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE"
> signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> {code}
> If we comment out the other tests, and only make testShouldFailButDoesnt run,
> then the build succeeds while it shouldn't. See the results in
> surefiretest/diagnostics/only_faulty_test/console_output.txt
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)
