Casey Jones created SUREFIRE-1970:
-------------------------------------
Summary: rerunFailingTestsCount option marks some spock tests as
false positive
Key: SUREFIRE-1970
URL: https://issues.apache.org/jira/browse/SUREFIRE-1970
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Plugin
Affects Versions: 3.0.0-M5, 3.0.0-M4
Reporter: Casey Jones
I have a project which uses [Spock Framework|https://spockframework.org/] for
tests. There is a single test which intentionally fails, no matter how many
times it was ran:
{code:java}
class FalsePositiveTest extends Specification {
def "test"() {
expect:
1 == number
where:
number << [2, 3]
}
} {code}
When test is executed with {{mvn test}} command, the build fails, as expected.
However, when using surefire's
[rerunFailingTestsCount|https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html]
option, the test in question is executed two times, as expected, but then it
is marked as flaky, and the build is successful:
{code:java}
mvn test -Dsurefire.rerunFailingTestsCount=1
...
Warning: Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
126[INFO]
127[INFO]
------------------------------------------------------------------------
128[INFO] BUILD SUCCESS
129[INFO]
------------------------------------------------------------------------{code}
Please check the links below for a full build log.
This issue happens only when using Spock's data-driven tests with a [where
block|https://spockframework.org/spock/docs/1.0/data_driven_testing.html] and
surefire's {{rerunFailingTestsCount}} option together. It seems like there is
something wrong in the way surefire detects if test has failed or passed.
I asked this [question on
StackOverflow|https://stackoverflow.com/questions/70454980/spock-data-driven-tests-surefire-rerunfailingtestscount-gives-false-positive-t]
and Spock's maintainer suggested to raise an issue in surefire's bugtracker.
Here is a simple github project which reproduces the issue:
[https://github.com/streetturtle/FalsePositiveTest]
Please check build logs of a github action here:
[https://github.com/streetturtle/FalsePositiveTest/runs/4613803891?check_suite_focus=true#step:4:128]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
