Hugo G created SUREFIRE-2276:
--------------------------------
Summary: JUnit5's TestTemplate failures treated as flakes with
retries
Key: SUREFIRE-2276
URL: https://issues.apache.org/jira/browse/SUREFIRE-2276
Project: Maven Surefire
Issue Type: Bug
Components: JUnit 5.x support, Maven Surefire Report Plugin
Affects Versions: 3.5.0
Reporter: Hugo G
G'day from Atlassian,
we found a bug in Surefire that creates false positive test results under
certain conditions.
TLDR: When `surefire.rerunFailingTestsCount` is set to 1 or higher, then test
template test failures are classified as flakes rather than failures.
Longer version:
JUnit5 has a feature called Test Templates
([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
which is very similar to the well known parametrised tests. The main
difference is that in a test template, we can not only provide parameters for
tests, but the entire context, including lifecycle methods (`@Before...`,
`@After...` etc.).
When one of the test method runs fails, Surefire correctly summarises the
failure and exits with a failure. E.g. if we provided the test template method
with two contexts, of which one reliably fails and one succeeds, then Surefire
would, just like JUnit, show that it had run two tests, of which one failed.
There is a problem with `surefire.rerunFailingTestsCount` though. The setting
correctly invokes retries on the right tests, but the final summary of failures
is wrong. With one working and one consistently failing test context, the
failing one would be re-run, but the final summary would say that we had only
ONE test and that it was flaky.
Interestingly, the problem does NOT surface when a parametrised test fails
consistently. There seems to be very specific logic for handling that, which
does not cover test templates.
Failing test case demonstrating the problem at
[https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
We are currently investigating further and trying to get a patch ready.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
