Ramanan Rajendran created SUREFIRE-2087:
-------------------------------------------

             Summary: rerunFailingTestsCount incorrectly marks failed 
parametrized test as pass
                 Key: SUREFIRE-2087
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-2087
             Project: Maven Surefire
          Issue Type: Bug
            Reporter: Ramanan Rajendran


rerunFailingTestsCount is not working as expected with junit5 
`@ParameterizedTest` 

If 1st iteration of the test passes and the 2nd one fails, it still marks the 
test as flaky passed. 

Eg: In the following run, iteration 1 with input A passed but iteration 2 with 
input B failed. Iteration 2 was rerun 2 times and failed in all the attempts. 
Instead of failing TestA, it's marked as flaky giving an overall pass for the 
test suite. I believe it's using the Run 1 PASS as a gate to mark the whole 
test as a pass. 

 
[INFO]  Run 1: PASS
[ERROR]  Run 2: TestA.iteration2:101 1 expectation failed.
Expected status code <400> doesn't match actual status code <401>.
[ERROR]  Run 3: TestA.iteration2:101 1 expectation failed.
Expected status code <400> doesn't match actual status code <401>.
[ERROR]  Run 4: TestA.iteration2:101 1 expectation failed.
 <400> doesn't match actual status code <401>.
 
Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Reply via email to