Daniel Fesenmeyer created SUREFIRE-2274:
-------------------------------------------
Summary: failOnFlakeCount does not work for failsafe
Key: SUREFIRE-2274
URL: https://issues.apache.org/jira/browse/SUREFIRE-2274
Project: Maven Surefire
Issue Type: Bug
Components: Maven Failsafe Plugin
Affects Versions: 3.5.0
Reporter: Daniel Fesenmeyer
The property failOnFlakeCount seems not to work for the failsafe plugin. At
least not as it is documented here:
[https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
h2. Expected behavior
[|https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer#expected-behavior]
When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd
expect that the build fails in case there are more than the specified count of
flakes.
h2. Actual behavior
[|https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer#actual-behavior]
The build succeeds, even when there are more than the specified count of
flakes. On the console, and the _TEST-***.xml_ report files, flakes are
correctly reported. However, the report {_}failsafe-summary.xml{_}, does not
contain a {{flakes}} property, and thus the verify step fails.
h2. How to reproduce
I've created a reproducer here:
[https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
Just run the integration tests for this simple maven project:
mvn verify
The build shows the following output, which is correct, except for the
information that the build succeeded (and status code 0):
{code}
...
[INFO] Running de.fesenmeyer.reproducer.ReproducerIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s
-- in de.fesenmeyer.reproducer.ReproducerIT
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
[ERROR] Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
[ERROR] Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[INFO]
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
[INFO]
[INFO]
[INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
...
{code}
h2. Hint
Probably the Feature currently works for Surefire, it seems like it has been
only tested with Surefire.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
