[
https://issues.apache.org/jira/browse/SUREFIRE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Soutter updated SUREFIRE-2144:
------------------------------------
Description:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
I suspect the fix is here is that Surefire should never create and pretend to
run synthetic tests, as they break the Jenkins reporting.
Rather, the entire run should fail - I presume throw an exception out of the
surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
was:
My team is dynamically creating tests using the JUnit3 suite api, and running
this using Jenkins CI
There is a nasty failure pattern which periodically resets the Age of all tests
in Jenkins back to 1. The Age report in Jenkins is key for us as it reveals
which commit caused the failure. We don't want to lose that information.
It goes like this
* many tests are running fine, with some failures
* a commit is made, CI triggers, Jenkins runs surefire.
** This results in a problem inside the suite() method, which throws a
RuntimeException. This is the dynamic test creation phase, before any tests are
run.
** This results in Surefire reporting a successful run of a single
"fake/synthetic" test which is reported as failed.
* a commit is made to fix the test creation phase, CI again triggeers, Jenkins
runs surefire
** This results in many tests again running fine, with the same failures as
before
** However, Jenkins now reports all the old failures from the first step as Age
1 - all the Age history is lost
The synthetic test failure looks like so:
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847 s
<<< FAILURE! - in com.company.package.MySuite
[ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046 s
<<< ERROR!
{code}
There is no "initializationError" test in our source code. I presume Surefire
has created it.
I think the fix is here is that Surefire should never create and pretend to run
synthetic tests, as they break the Jenkins reporting.
Rather, the entire run should fail - I presume throw an exception out of the
surefire plugin back to maven?
Hopefully this would then prevent Jenkins from doing the test analysis and it
will not break the history / Age reporting.
> Creating synthetic failures for initialization error breaks the Jenkins test
> history
> ------------------------------------------------------------------------------------
>
> Key: SUREFIRE-2144
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2144
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 2.22.2
> Reporter: Geoff Soutter
> Priority: Major
>
> My team is dynamically creating tests using the JUnit3 suite api, and running
> this using Jenkins CI
> There is a nasty failure pattern which periodically resets the Age of all
> tests in Jenkins back to 1. The Age report in Jenkins is key for us as it
> reveals which commit caused the failure. We don't want to lose that
> information.
> It goes like this
> * many tests are running fine, with some failures
> * a commit is made, CI triggers, Jenkins runs surefire.
> ** This results in a problem inside the suite() method, which throws a
> RuntimeException. This is the dynamic test creation phase, before any tests
> are run.
> ** This results in Surefire reporting a successful run of a single
> "fake/synthetic" test which is reported as failed.
> * a commit is made to fix the test creation phase, CI again triggeers,
> Jenkins runs surefire
> ** This results in many tests again running fine, with the same failures as
> before
> ** However, Jenkins now reports all the old failures from the first step as
> Age 1 - all the Age history is lost
> The synthetic test failure looks like so:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.847
> s <<< FAILURE! - in com.company.package.MySuite
> [ERROR] initializationError(com.company.package.MySuite) Time elapsed: 0.046
> s <<< ERROR!
> {code}
> There is no "initializationError" test in our source code. I presume Surefire
> has created it.
> I suspect the fix is here is that Surefire should never create and pretend to
> run synthetic tests, as they break the Jenkins reporting.
> Rather, the entire run should fail - I presume throw an exception out of the
> surefire plugin back to maven?
> Hopefully this would then prevent Jenkins from doing the test analysis and it
> will not break the history / Age reporting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
