[
https://issues.apache.org/jira/browse/SUREFIRE-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hugo G updated SUREFIRE-2276:
-----------------------------
Description:
G'day from Atlassian,
*TLDR:* When {{surefire.rerunFailingTestsCount}} is set to 1 or higher, then
test template test failures are classified as flakes rather than failures.
Failing test case demonstrating the problem at
[https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
JUnit5 has a feature called Test Templates
([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
which is very similar to the well known parametrised tests. The main
difference is that in a test template, we can not only provide parameters for
tests, but the entire context, including lifecycle methods ({{{}@Before...{}}},
{{@After...}} etc.).
When one of the test method runs fails, Surefire correctly summarises the
failure and exits with a failure. E.g. if we provided the test template method
with two contexts, of which one reliably fails and one succeeds, then Surefire
would, just like JUnit, show that it had run two tests, of which one failed.
There is a problem with {{surefire.rerunFailingTestsCount}} though. The setting
correctly causes Surefire to invoke retries on the right tests, but the final
summary of failures is wrong. With one working and one consistently failing
test context, the failing one would be re-run, but the final summary would say
that we had only ONE test and that it was flaky. This causes Surefire to exit
with a success code, even though tests did fail.
Interestingly, the problem does NOT surface when a parametrised test fails
consistently, even though parametrised tests use test templates internally.
There seems to be very specific logic for handling parametrised tests, which
does not cover test templates around here:
[https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325.|https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325]
The test case in
[https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
shows how adding the retries changes the test overall outcome and the summary
stats.
Expected: 2 run, 1 failure. Overall: fail
Actual: 1 run, 1 flake. Overall: success
We are currently investigating further and trying to get a patch ready.
was:
G'day from Atlassian,
*TLDR:* When {{surefire.rerunFailingTestsCount}} is set to 1 or higher, then
test template test failures are classified as flakes rather than failures.
Failing test case demonstrating the problem at
[https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
JUnit5 has a feature called Test Templates
([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
which is very similar to the well known parametrised tests. The main
difference is that in a test template, we can not only provide parameters for
tests, but the entire context, including lifecycle methods ({{{}@Before...{}}},
{{@After...}} etc.).
When one of the test method runs fails, Surefire correctly summarises the
failure and exits with a failure. E.g. if we provided the test template method
with two contexts, of which one reliably fails and one succeeds, then Surefire
would, just like JUnit, show that it had run two tests, of which one failed.
There is a problem with {{surefire.rerunFailingTestsCount}} though. The setting
correctly causes Surefire to invoke retries on the right tests, but the final
summary of failures is wrong. With one working and one consistently failing
test context, the failing one would be re-run, but the final summary would say
that we had only ONE test and that it was flaky. This causes Surefire to exit
with a success code, even though tests did fail.
Interestingly, the problem does NOT surface when a parametrised test fails
consistently, even though parametrised tests use test templates internally.
There seems to be very specific logic for handling parametrised tests, which
does not cover test templates around here:
[https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325.|https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325]
We are currently investigating further and trying to get a patch ready.
> JUnit5's TestTemplate failures treated as flakes with retries
> -------------------------------------------------------------
>
> Key: SUREFIRE-2276
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2276
> Project: Maven Surefire
> Issue Type: Bug
> Components: JUnit 5.x support, Maven Surefire Report Plugin
> Affects Versions: 3.5.0
> Reporter: Hugo G
> Priority: Minor
>
> G'day from Atlassian,
> *TLDR:* When {{surefire.rerunFailingTestsCount}} is set to 1 or higher, then
> test template test failures are classified as flakes rather than failures.
> Failing test case demonstrating the problem at
> [https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
> JUnit5 has a feature called Test Templates
> ([docs|https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-templates])
> which is very similar to the well known parametrised tests. The main
> difference is that in a test template, we can not only provide parameters for
> tests, but the entire context, including lifecycle methods
> ({{{}@Before...{}}}, {{@After...}} etc.).
> When one of the test method runs fails, Surefire correctly summarises the
> failure and exits with a failure. E.g. if we provided the test template
> method with two contexts, of which one reliably fails and one succeeds, then
> Surefire would, just like JUnit, show that it had run two tests, of which one
> failed.
> There is a problem with {{surefire.rerunFailingTestsCount}} though. The
> setting correctly causes Surefire to invoke retries on the right tests, but
> the final summary of failures is wrong. With one working and one consistently
> failing test context, the failing one would be re-run, but the final summary
> would say that we had only ONE test and that it was flaky. This causes
> Surefire to exit with a success code, even though tests did fail.
> Interestingly, the problem does NOT surface when a parametrised test fails
> consistently, even though parametrised tests use test templates internally.
> There seems to be very specific logic for handling parametrised tests, which
> does not cover test templates around here:
> [https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325.|https://github.com/apache/maven-surefire/blob/d7f4dbb7cc99a08a911066afbab3e01273dbbdc0/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L325]
>
> The test case in
> [https://github.com/apache/maven-surefire/compare/master...atlassian-forks:maven-surefire:master]
> shows how adding the retries changes the test overall outcome and the
> summary stats.
> Expected: 2 run, 1 failure. Overall: fail
> Actual: 1 run, 1 flake. Overall: success
>
> We are currently investigating further and trying to get a patch ready.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
