Benoit Lagae created SUREFIRE-1113:
--------------------------------------
Summary: Build does not fail when successPercentage for
@org.testng.annotations.Test() is not met
Key: SUREFIRE-1113
URL: https://jira.codehaus.org/browse/SUREFIRE-1113
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Plugin, TestNG support
Affects Versions: 2.18
Environment: TestNG 6.8.8, maven-surefire-plugin 2.19-SNAPSHOT
(53a40eef48ea)
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4;
2014-08-11T22:58:10+02:00)
Maven home: C:\Program Files\apache-maven-3.2.3
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.7.0_67\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
Reporter: Benoit Lagae
Attachments: surefiretest.zip
When a TestNG test with annotation methods invocationCount = x and
successPercentage = y fails, this does not break the Maven build of the
project. I have attached a minimal Maven project with only the pom and a single
test class, which contains three dummy tests which should all fail:
@Test
public void testFailsCorrectly() { fail("dummy"); }
@Test(invocationCount = x)
public void testCounterWorksAndTestFails() { fail("dummy"); }
@Test(invocationCount = x, successPercentage = y)
public void testShouldFailButDoesnt() { fail("dummy"); }
The first two comply, but the third one doesn't, as is implied in the name of
the method. I created this project from scratch, i.e. not from an IDE: manual
creation of folders, code and pom in Notepad++, ...
I have attached both the project and the info I thought could be relevant from
my test runs of the project ('diagnostics' folder). Below are the results of
the individual method runs:
<test-method status="FAIL" signature="testCounterWorks()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
<test-method status="FAIL" signature="testCounterWorks()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
<test-method status="FAIL" signature="testCounterWorks()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
<test-method status="FAIL" signature="testCounterWorks()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
<test-method status="FAIL" signature="testFailsCorrectly()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testFailsCorrectly"/>
<test-method status="SUCCESS_PERCENTAGE_FAILURE"
signature="testShouldFailButDoesnt()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
<test-method status="SUCCESS_PERCENTAGE_FAILURE"
signature="testShouldFailButDoesnt()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
<test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
<test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
If we comment out the other tests, and only make testShouldFailButDoesnt run,
then the build succeeds while it shouldn't. See the results in
surefiretest/diagnostics/only_faulty_test/console_output.txt
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)
