[ https://issues.apache.org/jira/browse/SUREFIRE-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645399#comment-17645399 ]
Basil Crow commented on SUREFIRE-2087: -------------------------------------- The difference in behavior between the {{expected}} and {{actual}} modules of [https://github.com/basil/SUREFIRE-2087-mre] says it all. We have been impacted by this in the Jenkins project: after switching from {{org.apache.maven.surefire.junit4.JUnit4Provider}} to {{org.apache.maven.surefire.junitplatform.JUnitPlatformProvider}} for our (predominantly JUnit 4 based) test suite, someone made a change that broke a test. But Maven erroneously classified it as a flaky test and marked the overall run as successful, which caused the bug to go unnoticed. Later on, that bug resulted in a security vulnerability which required an emergency release to correct. So this is a very real problem that has affected us. We worked around the issue first by removing parameterization from the test in question, then by disabling {{rerunFailingTestsCount}} entirely, the latter of which forced us to spend months making our test suite less flaky. > rerunFailingTestsCount incorrectly marks failed parametrized test as pass > ------------------------------------------------------------------------- > > Key: SUREFIRE-2087 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2087 > Project: Maven Surefire > Issue Type: Bug > Affects Versions: 3.0.0-M4 > Reporter: Ramanan Rajendran > Priority: Critical > > rerunFailingTestsCount is not working as expected with junit5 > `@ParameterizedTest` > If 1st iteration of the test passes and the 2nd one fails, it still marks the > test as flaky passed. > Eg: In the following run, iteration 1 with input A passed but iteration 2 > with input B failed. Iteration 2 was rerun 2 times and failed in all the > attempts. Instead of failing TestA, it's marked as flaky giving an overall > pass for the test suite. I believe it's using the Run 1 PASS as a gate to > mark the whole test as a pass. > > [INFO] Run 1: PASS > [ERROR] Run 2: TestA.iteration2:101 1 expectation failed. > Expected status code <400> doesn't match actual status code <401>. > [ERROR] Run 3: TestA.iteration2:101 1 expectation failed. > Expected status code <400> doesn't match actual status code <401>. > [ERROR] Run 4: TestA.iteration2:101 1 expectation failed. > <400> doesn't match actual status code <401>. > > Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1 -- This message was sent by Atlassian Jira (v8.20.10#820010)