[
https://jira.codehaus.org/browse/SUREFIRE-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=355609#comment-355609
]
Benoit Lagae commented on SUREFIRE-1113:
----------------------------------------
I have just tested this with 4 versions to demonstrate that this bug is indeed
related to recent developments, probably related to the very welcome fix for
SUREFIRE-654. Parameters for my test method are invocationCount = 4,
successPercentage = 70, and I only let the "testShouldFailButDoesnt" run so the
build could succeed. The only difference between the runs is the <version> node
for maven-surefire-plugin in my POM. Expected behavior is test failure and
consequently build failure.
2.16: "testShouldFailButDoesnt" gives 4 errors - not test failures, as for the
other methods when I let them run, which correctly fail.
{code}
testShouldFailButDoesnt(blagae.StatisticsTest) Time elapsed: 0 sec <<< ERROR!
java.lang.AssertionError: dummy
at org.testng.Assert.fail(Assert.java:94)
at blagae.StatisticsTest.testShouldFailButDoesnt(StatisticsTest.java:20)
{code}
2.17: Identical to 2.16
2.18 (from Maven-Central): "testShouldFailButDoesnt" passes, build succeeds
2.19-SNAPSHOT (own build from commit 53a40eef48ea): "testShouldFailButDoesnt"
passes, build succeeds
The issue in SUREFIRE-654 was that the SureFire plugin gave false negatives
i.e. it didn't allow for acceptable incidental test failure as implemented by
TestNG. Now, on the other hand, it seems to give false positives i.e. it
doesn't pick up unacceptable frequency of test failure.
> Build does not fail when successPercentage for @org.testng.annotations.Test()
> is not met
> ----------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1113
> URL: https://jira.codehaus.org/browse/SUREFIRE-1113
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin, TestNG support
> Affects Versions: 2.18
> Environment: TestNG 6.8.8, maven-surefire-plugin 2.19-SNAPSHOT
> (53a40eef48ea)
> Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4;
> 2014-08-11T22:58:10+02:00)
> Maven home: C:\Program Files\apache-maven-3.2.3
> Java version: 1.7.0_67, vendor: Oracle Corporation
> Java home: C:\Program Files\Java\jdk1.7.0_67\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
> Reporter: Benoit Lagae
> Attachments: surefiretest.zip
>
>
> When a TestNG test with annotation methods invocationCount = x and
> successPercentage = y fails, this does not break the Maven build of the
> project. I have attached a minimal Maven project with only the pom and a
> single test class, which contains three dummy tests which should all fail:
> {code}
> @Test
> public void testFailsCorrectly() { fail("dummy"); }
> @Test(invocationCount = x)
> public void testCounterWorksAndTestFails() { fail("dummy"); }
> @Test(invocationCount = x, successPercentage = y)
> public void testShouldFailButDoesnt() { fail("dummy"); }
> {code}
> The first two comply, but the third one doesn't, as is implied in the name of
> the method. I created this project from scratch, i.e. not from an IDE: manual
> creation of folders, code and pom in Notepad++, ...
> I have attached both the project and the info I thought could be relevant
> from my test runs of the project ('diagnostics' folder). Below are the
> results of the individual method runs:
> {code}
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testCounterWorks()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testCounterWorks"/>
> <test-method status="FAIL" signature="testFailsCorrectly()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testFailsCorrectly"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE"
> signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="SUCCESS_PERCENTAGE_FAILURE"
> signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> <test-method status="FAIL" signature="testShouldFailButDoesnt()[pri:0,
> instance:blagae.StatisticsTest@575598a]" name="testShouldFailButDoesnt"/>
> {code}
> If we comment out the other tests, and only make testShouldFailButDoesnt run,
> then the build succeeds while it shouldn't. See the results in
> surefiretest/diagnostics/only_faulty_test/console_output.txt
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)
