Sean Flanigan created SUREFIRE-1219:
---------------------------------------
Summary: skipAfterFailureCount should not count flaky tests as
failures when using rerunFailingTestsCount
Key: SUREFIRE-1219
URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
Project: Maven Surefire
Issue Type: Bug
Affects Versions: 2.19.1
Reporter: Sean Flanigan
According to
https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
"failed tests within re-run phase are not included in skipAfterFailureCount".
For instance, if skipAfterFailureCount is 1, but every test passes on the
second attempt, the tests are "flaky" but not "failures". This should be
reflected in summaries, in XML result files and in fail fast
(skipAfterFailureCount), but this is not the case.
If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and
skipAfterFailureCount=5:
1. I would expect something like this to be printed on the console:
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10
2. The XML report should list <flakyFailure>s but no <failures>.
3. All tests should be run, not just 5 of them, in other words
skipAfterFailureCount should not count the flaky tests as part of the failure
count.
I have some tests demonstrating the problem, which I will attach to this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)