[
https://issues.apache.org/jira/browse/SUREFIRE-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Fesenmeyer updated SUREFIRE-2274:
----------------------------------------
Description:
The property failOnFlakeCount seems not to work for the failsafe plugin. At
least not as it is documented here:
[https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
h2. Expected behavior
When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd
expect that the build fails in case there are more than the specified count of
flakes.
h2. Actual behavior
The build succeeds, even when there are more than the specified count of
flakes. On the console, and the _TEST-***.xml_ report files, flakes are
correctly reported. However, the report {_}failsafe-summary.xml{_}, does not
contain a {{flakes}} property, and thus the verify step fails.
h2. How to reproduce
I've created a reproducer here:
[https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
Just run the integration tests for this simple maven project:
mvn verify
The build shows the following output, which is correct, except for the
information that the build succeeded (and status code 0):
{code:java}
...
[INFO] Running de.fesenmeyer.reproducer.ReproducerIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s
-- in de.fesenmeyer.reproducer.ReproducerIT
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
[ERROR] Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
[ERROR] Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[INFO]
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
[INFO]
[INFO]
[INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
...
{code}
h2. Hint
Probably the Feature currently works for Surefire, it seems like it has been
only tested with Surefire.
was:
The property failOnFlakeCount seems not to work for the failsafe plugin. At
least not as it is documented here:
[https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
h2. Expected behavior
[|https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer#expected-behavior]
When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd
expect that the build fails in case there are more than the specified count of
flakes.
h2. Actual behavior
[|https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer#actual-behavior]
The build succeeds, even when there are more than the specified count of
flakes. On the console, and the _TEST-***.xml_ report files, flakes are
correctly reported. However, the report {_}failsafe-summary.xml{_}, does not
contain a {{flakes}} property, and thus the verify step fails.
h2. How to reproduce
I've created a reproducer here:
[https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
Just run the integration tests for this simple maven project:
mvn verify
The build shows the following output, which is correct, except for the
information that the build succeeded (and status code 0):
{code}
...
[INFO] Running de.fesenmeyer.reproducer.ReproducerIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s
-- in de.fesenmeyer.reproducer.ReproducerIT
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
[ERROR] Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
[ERROR] Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[INFO]
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
[INFO]
[INFO]
[INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
...
{code}
h2. Hint
Probably the Feature currently works for Surefire, it seems like it has been
only tested with Surefire.
> failOnFlakeCount does not work for failsafe
> -------------------------------------------
>
> Key: SUREFIRE-2274
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2274
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Failsafe Plugin
> Affects Versions: 3.5.0
> Reporter: Daniel Fesenmeyer
> Priority: Major
>
> The property failOnFlakeCount seems not to work for the failsafe plugin. At
> least not as it is documented here:
> [https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
> h2. Expected behavior
> When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd
> expect that the build fails in case there are more than the specified count
> of flakes.
> h2. Actual behavior
> The build succeeds, even when there are more than the specified count of
> flakes. On the console, and the _TEST-***.xml_ report files, flakes are
> correctly reported. However, the report {_}failsafe-summary.xml{_}, does not
> contain a {{flakes}} property, and thus the verify step fails.
> h2. How to reproduce
> I've created a reproducer here:
> [https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
> Just run the integration tests for this simple maven project:
> mvn verify
> The build shows the following output, which is correct, except for the
> information that the build succeeded (and status code 0):
> {code:java}
> ...
> [INFO] Running de.fesenmeyer.reproducer.ReproducerIT
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003
> s -- in de.fesenmeyer.reproducer.ReproducerIT
> [INFO]
> [INFO] Results:
> [INFO]
> [WARNING] Flakes:
> [WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
> [ERROR] Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
> [ERROR] Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
> [INFO] Run 3: PASS
> [INFO]
> [WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
> [ERROR] Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
> [ERROR] Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
> [INFO] Run 3: PASS
> [INFO]
> [INFO]
> [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
> [INFO]
> [INFO]
> [INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer
> ---
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> ...
> {code}
> h2. Hint
> Probably the Feature currently works for Surefire, it seems like it has been
> only tested with Surefire.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
