[
https://issues.apache.org/jira/browse/SUREFIRE-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Osipov reassigned SUREFIRE-2276:
----------------------------------------
Assignee: Michael Osipov
> JUnit5's TestTemplate failures treated as flakes with retries
> -------------------------------------------------------------
>
> Key: SUREFIRE-2276
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2276
> Project: Maven Surefire
> Issue Type: Bug
> Components: JUnit 5.x support, Maven Surefire Report Plugin
> Affects Versions: 3.5.0
> Reporter: Hugo G
> Assignee: Michael Osipov
> Priority: Minor
>
> G'day from Atlassian,
> *TLDR:* When {{surefire.rerunFailingTestsCount}} is set to 1 or higher, then
> test template test failures are classified as flakes rather than failures.
> Failing test case demonstrating the problem at
> [https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
> JUnit5 has a feature called Test Templates
> ([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
> which is very similar to the well known parametrised tests. The main
> difference is that in a test template, we can not only provide parameters for
> tests, but the entire context, including lifecycle methods
> ({{{}@Before...{}}}, {{@After...}} etc.).
> When one of the test method runs fails, Surefire correctly summarises the
> failure and exits with a failure. E.g. if we provided the test template
> method with two contexts, of which one reliably fails and one succeeds, then
> Surefire would, just like JUnit, show that it had run two tests, of which one
> failed.
> There is a problem with {{surefire.rerunFailingTestsCount}} though. The
> setting correctly causes Surefire to invoke retries on the right tests, but
> the final summary of failures is wrong. With one working and one consistently
> failing test context, the failing one would be re-run, but the final summary
> would say that we had only ONE test and that it was flaky. This causes
> Surefire to exit with a success code, even though tests did fail.
> Interestingly, the problem does NOT surface when a parametrised test fails
> consistently, even though parametrised tests use test templates internally.
> There seems to be very specific logic for handling parametrised tests, which
> does not cover test templates around here:
> [https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325.|https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325]
>
> The test case in
> [https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
> shows how adding the retries changes the test overall outcome and the
> summary stats.
> Expected: 2 run, 1 failure. Overall: fail
> Actual: 1 run, 1 flake. Overall: success
> I also added a test class for parametrised tests, where one param makes the
> test consistently fail. The test class is commented out in the IT, but might
> help with debugging.
>
> We are currently investigating further and trying to get a patch ready.
> EDIT: We have a patch ready for merging at
> [https://github.com/apache/maven-surefire/pull/788]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
