[
https://issues.apache.org/jira/browse/SUREFIRE-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645466#comment-17645466
]
Basil Crow commented on SUREFIRE-2087:
--------------------------------------
bq. So this works as you expect in JUnit 4 but not JUnit 5?
Yes, as I noted in the very first comment:
bq. Note that this bug even affects JUnit 4 based parameterized tests as long
as they are executed with JUnit 5 Vintage rather than JUnit 4.
bq. {{rerunFailingTests}} is something of a hack, and making tests less flaky
is an appropriate response.
It is not deprecated in the documentation, and certainly worked fine in the
past with {{org.apache.maven.surefire.junit4.JUnit4Provider}}. Putting the
Jenkins core development team under unexpected pressure to make its test suite
less flaky (or revert back to
{{org.apache.maven.surefire.junit4.JUnit4Provider}}) was certainly not viewed
favorably.
> rerunFailingTestsCount incorrectly marks failed parametrized test as pass
> -------------------------------------------------------------------------
>
> Key: SUREFIRE-2087
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2087
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M4
> Reporter: Ramanan Rajendran
> Priority: Critical
>
> rerunFailingTestsCount is not working as expected with junit5
> `@ParameterizedTest`
> If 1st iteration of the test passes and the 2nd one fails, it still marks the
> test as flaky passed.
> Eg: In the following run, iteration 1 with input A passed but iteration 2
> with input B failed. Iteration 2 was rerun 2 times and failed in all the
> attempts. Instead of failing TestA, it's marked as flaky giving an overall
> pass for the test suite. I believe it's using the Run 1 PASS as a gate to
> mark the whole test as a pass.
>
> [INFO] Run 1: PASS
> [ERROR] Run 2: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 3: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 4: TestA.iteration2:101 1 expectation failed.
> <400> doesn't match actual status code <401>.
>
> Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
