[
https://issues.apache.org/jira/browse/SUREFIRE-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hugo G updated SUREFIRE-2276:
-----------------------------
Description:
G'day from Atlassian,
we found a bug in Surefire that creates false positive test results under
certain conditions.
TLDR: When `surefire.rerunFailingTestsCount` is set to 1 or higher, then test
template test failures are classified as flakes rather than failures.
Longer version:
JUnit5 has a feature called Test Templates
([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
which is very similar to the well known parametrised tests. The main
difference is that in a test template, we can not only provide parameters for
tests, but the entire context, including lifecycle methods (`@Before...`,
`@After...` etc.).
When one of the test method runs fails, Surefire correctly summarises the
failure and exits with a failure. E.g. if we provided the test template method
with two contexts, of which one reliably fails and one succeeds, then Surefire
would, just like JUnit, show that it had run two tests, of which one failed.
There is a problem with `surefire.rerunFailingTestsCount` though. The setting
correctly invokes retries on the right tests, but the final summary of failures
is wrong. With one working and one consistently failing test context, the
failing one would be re-run, but the final summary would say that we had only
ONE test and that it was flaky.
Interestingly, the problem does NOT surface when a parametrised test fails
consistently, even though parametrised tests use test templates internally.
There seems to be very specific logic for handling parametrised tests, which
does not cover test templates around here:
[https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325.|https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325]
Failing test case demonstrating the problem at
[https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
We are currently investigating further and trying to get a patch ready.
was:
G'day from Atlassian,
we found a bug in Surefire that creates false positive test results under
certain conditions.
TLDR: When `surefire.rerunFailingTestsCount` is set to 1 or higher, then test
template test failures are classified as flakes rather than failures.
Longer version:
JUnit5 has a feature called Test Templates
([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
which is very similar to the well known parametrised tests. The main
difference is that in a test template, we can not only provide parameters for
tests, but the entire context, including lifecycle methods (`@Before...`,
`@After...` etc.).
When one of the test method runs fails, Surefire correctly summarises the
failure and exits with a failure. E.g. if we provided the test template method
with two contexts, of which one reliably fails and one succeeds, then Surefire
would, just like JUnit, show that it had run two tests, of which one failed.
There is a problem with `surefire.rerunFailingTestsCount` though. The setting
correctly invokes retries on the right tests, but the final summary of failures
is wrong. With one working and one consistently failing test context, the
failing one would be re-run, but the final summary would say that we had only
ONE test and that it was flaky.
Interestingly, the problem does NOT surface when a parametrised test fails
consistently, even though parametrised tests use test templates internally.
There seems to be very specific logic for handling parametrised tests, which
does not cover test templates.
Failing test case demonstrating the problem at
[https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
We are currently investigating further and trying to get a patch ready.
> JUnit5's TestTemplate failures treated as flakes with retries
> -------------------------------------------------------------
>
> Key: SUREFIRE-2276
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2276
> Project: Maven Surefire
> Issue Type: Bug
> Components: JUnit 5.x support, Maven Surefire Report Plugin
> Affects Versions: 3.5.0
> Reporter: Hugo G
> Priority: Minor
>
> G'day from Atlassian,
> we found a bug in Surefire that creates false positive test results under
> certain conditions.
>
> TLDR: When `surefire.rerunFailingTestsCount` is set to 1 or higher, then test
> template test failures are classified as flakes rather than failures.
>
> Longer version:
> JUnit5 has a feature called Test Templates
> ([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
> which is very similar to the well known parametrised tests. The main
> difference is that in a test template, we can not only provide parameters for
> tests, but the entire context, including lifecycle methods (`@Before...`,
> `@After...` etc.).
> When one of the test method runs fails, Surefire correctly summarises the
> failure and exits with a failure. E.g. if we provided the test template
> method with two contexts, of which one reliably fails and one succeeds, then
> Surefire would, just like JUnit, show that it had run two tests, of which one
> failed.
> There is a problem with `surefire.rerunFailingTestsCount` though. The setting
> correctly invokes retries on the right tests, but the final summary of
> failures is wrong. With one working and one consistently failing test
> context, the failing one would be re-run, but the final summary would say
> that we had only ONE test and that it was flaky.
> Interestingly, the problem does NOT surface when a parametrised test fails
> consistently, even though parametrised tests use test templates internally.
> There seems to be very specific logic for handling parametrised tests, which
> does not cover test templates around here:
> [https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325.|https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325]
> Failing test case demonstrating the problem at
> [https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
>
> We are currently investigating further and trying to get a patch ready.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
