[
https://jira.codehaus.org/browse/SUREFIRE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=354999#comment-354999
]
Andreas Gudian commented on SUREFIRE-654:
-----------------------------------------
I'm afraid I found what's wrong.
The test run listener reports all test attempts, the failing and the successful
and they all show up in the test results / reports. So far so good, I guess.
But at the end, Surefire checks for each test method, if it was executed
multiple times. If a test was executed multiple times and there were both
failing and successfull attempts (regardless of their actual number), the test
is marked as "Flaky" (a new feature introduced in 2.18):
{code}
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.265 sec <<<
FAILURE! - in TestSuite
testFailure(testng.TestNGSuccessPercentFailingTest) Time elapsed: 0 sec <<<
FAILURE!
java.lang.AssertionError: is odd: 3 expected:<true> but was:<false>
at org.testng.Assert.fail(Assert.java:84)
at org.testng.Assert.failNotEquals(Assert.java:438)
at org.testng.Assert.assertTrue(Assert.java:32)
at
testng.TestNGSuccessPercentFailingTest.testFailure(TestNGSuccessPercentFailingTest.java:36)
Results :
Flaked tests:
testng.TestNGSuccessPercentFailingTest.testFailure(testng.TestNGSuccessPercentFailingTest)
Run 1: TestNGSuccessPercentFailingTest.testFailure:36 is odd: 1
expected:<true> but was:<false>
Run 2: PASS
Run 3: TestNGSuccessPercentFailingTest.testFailure:36 is odd: 3
expected:<true> but was:<false>
Run 4: PASS
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
{code}
Now Andrew's test contains this:
{code}
@Test(invocationCount = 4, successPercentage = 99)
public void testShouldFail_percentage99() {
assertTrue(false);
}
{code}
The first three times, the test is reported by TestNG via
"onTestFailedButWithinSuccessPercentage", and with the patch marked as
"success" in surefire. The last failure is reported via "onTestFailure", the
new flaky-algorithm decides then that the test is just "flaky".
I've changed the flaky-detection to be only active if the new surefire property
rerunFailingTestsCount > 0.
Let's see if that helps.
> Surefire does not recognize successful tests using TestNG with the
> invocationCount and successPercentage parameters on the @Test annotation
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-654
> URL: https://jira.codehaus.org/browse/SUREFIRE-654
> Project: Maven Surefire
> Issue Type: Bug
> Components: TestNG support
> Affects Versions: 2.6
> Environment: Apache Maven 2.2.1 (rdebian-4)
> Java version: 1.6.0-18
> Java home: /usr/lib/jvm/java-6-openjdk/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux" version: "2.6.35-22-generic" arch: "amd64" Family: "unix"
> Ubuntu 10.10
> Reporter: Steve Stodola
> Assignee: Andreas Gudian
> Fix For: 2.18
>
> Attachments: bugtest.tar.gz, surefire654.diff,
> surefire-fix-test-build-output.txt, surefire-fix-test.zip
>
>
> Surefire does not recognize successful tests using TestNG with the
> invocationCount and successPercentage parameters on the @Test annotation.
> BugTest.java in the attached file has an invocationCount = 100 with a
> successPercentage = 1. 1 test in 100 needs to pass for the test to be
> considered successful. However when running *{{mvn clean test}}* Surefire
> reports the tests as failed. The generated report shows the tests as passing.
> Surefire should be identifying this test as passing.
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)
