[
https://issues.apache.org/jira/browse/SUREFIRE-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated SUREFIRE-2039:
-------------------------------------
Description:
The setup: maven-surefire-plugin 3.0.0-M5 and junit 5.8.2.
Consider a simple scenario, two test classes, *ATest* and *BTest*, one
+Disabled+ and one not respectively, each has three test methods. After running
_mvn surefire:test_ the report will indicate that 4 tests were run and one of
them was skipped:
{noformat}
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.test.BTest
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.062
s - in org.example.test.BTest
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1
{noformat}
And if we enable *ATest* and disable *BTest*, then the report will only mention
running three tests, none skipped:
{noformat}
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.test.ATest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 s
- in org.example.test.ATest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
{noformat}
This behavior is the result of the way how
*org.apache.maven.plugin.surefire.report.TestSetRunListener* handles skipped
tests. It adds them to the *detailsForThis* and only prints after the test set
is finished which happens when a non-disabled test set is finished thus adding
skipped test results to the statistics of a non-skipped test set. And, of
course, if the disabled test is ran after the non-disabled test, it won't be
reported at all.
The reproducer can be found here:
https://github.com/SammyVimesFiledIssues/SUREFIRE-2039
was:
Consider a simple scenario, two test classes, *ATest* and *BTest*, one
+Disabled+ and one not respectively, each has three test methods. After running
_mvn surefire:test_ the report will indicate that 4 tests were run and one of
them was skipped:
{noformat}
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.test.BTest
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.062
s - in org.example.test.BTest
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1
{noformat}
And if we enable *ATest* and disable *BTest*, then the report will only mention
running three tests, none skipped:
{noformat}
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.test.ATest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 s
- in org.example.test.ATest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
{noformat}
This behavior is the result of the way how
*org.apache.maven.plugin.surefire.report.TestSetRunListener* handles skipped
tests. It adds them to the *detailsForThis* and only prints after the test set
is finished which happens when a non-disabled test set is finished thus adding
skipped test results to the statistics of a non-skipped test set. And, of
course, if the disabled test is ran after the non-disabled test, it won't be
reported at all.
The reproducer can be found here:
https://github.com/SammyVimesFiledIssues/SUREFIRE-2039
> Skipped test classes are getting into the non-skipped test classes reports
> --------------------------------------------------------------------------
>
> Key: SUREFIRE-2039
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2039
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M5
> Reporter: Semyon Danilov
> Priority: Major
>
> The setup: maven-surefire-plugin 3.0.0-M5 and junit 5.8.2.
> Consider a simple scenario, two test classes, *ATest* and *BTest*, one
> +Disabled+ and one not respectively, each has three test methods. After
> running _mvn surefire:test_ the report will indicate that 4 tests were run
> and one of them was skipped:
> {noformat}
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.example.test.BTest
> [WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
> 0.062 s - in org.example.test.BTest
> [INFO]
> [INFO] Results:
> [INFO]
> [WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1
> {noformat}
> And if we enable *ATest* and disable *BTest*, then the report will only
> mention running three tests, none skipped:
> {noformat}
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.example.test.ATest
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079
> s - in org.example.test.ATest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> This behavior is the result of the way how
> *org.apache.maven.plugin.surefire.report.TestSetRunListener* handles skipped
> tests. It adds them to the *detailsForThis* and only prints after the test
> set is finished which happens when a non-disabled test set is finished thus
> adding skipped test results to the statistics of a non-skipped test set. And,
> of course, if the disabled test is ran after the non-disabled test, it won't
> be reported at all.
> The reproducer can be found here:
> https://github.com/SammyVimesFiledIssues/SUREFIRE-2039
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
