Sean Flanigan created SUREFIRE-1219:
---------------------------------------

             Summary: skipAfterFailureCount should not count flaky tests as 
failures when using rerunFailingTestsCount
                 Key: SUREFIRE-1219
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
             Project: Maven Surefire
          Issue Type: Bug
    Affects Versions: 2.19.1
            Reporter: Sean Flanigan


According to 
https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
 "failed tests within re-run phase are not included in skipAfterFailureCount". 
For instance, if skipAfterFailureCount is 1, but every test passes on the 
second attempt, the tests are "flaky" but not "failures".  This should be 
reflected in summaries, in XML result files and in fail fast 
(skipAfterFailureCount), but this is not the case.

If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and 
skipAfterFailureCount=5:

1. I would expect something like this to be printed on the console:
  Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10

2. The XML report should list <flakyFailure>s but no <failures>.

3. All tests should be run, not just 5 of them, in other words 
skipAfterFailureCount should not count the flaky tests as part of the failure 
count.

I have some tests demonstrating the problem, which I will attach to this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to