[
https://issues.apache.org/jira/browse/SUREFIRE-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
M.P. Korstanje updated SUREFIRE-1691:
-------------------------------------
Description:
This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- [https://github.com/cucumber/cucumber-jvm/issues/865]
The latest reproducer can be found here:
-
[https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
In short Cucumber has a hierarchy of tests that looks at follows:
{noformat}
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
{noformat}
Surefire tries to group output in tests sets, either per class or per test
suite. As Cucumber currently only emits test started/finished events for
Scenarios sure fire can not group scenarios into a test set. They are instead
grouped with the previous test set. However even after adding test
started/finished events for features, sure fire groups all results in a single
test set.
I believe this is caused by an error in
{{NonConcurrentRunListener.describesNewTestSet}}.
{code:java}
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals(
currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals(
currentTestSetDescription.getChildren() );
}
return false;
}
return true;
}
{code}
The value of {{description.getChildren().equals(
currentTestSetDescription.getChildren() );}} should be negated so so different
test suits are properly recognized as test sets.
Fixing this locally works and can be reproduced by:
1. Negating the line and building sure fire locally
2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on
[https://github.com/mpkorstanje/surefire-test]
It does however does result in another unwanted result: A feature file with two
scenarios is now reported as having 3 results. One for each scenario and one
for the feature.
{noformat}
-------------------------------------------------------------------------------
Test set: Some Feature 1
-------------------------------------------------------------------------------
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in
Some_Feature_1
{noformat}
was:
This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- [https://github.com/cucumber/cucumber-jvm/issues/865]
The latest reproducer can be found here:
-
[https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
In short Cucumber has a hierarchy of tests that looks at follows:
{noformat}
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
{noformat}
Surefire tries to group output in tests sets, either per class or per test
suite. As Cucumber currently only emits test started/finished events for
Scenarios sure fire can not group scenarios into a test set. They are instead
group with the previous test set. However even after adding test
started/finished events for features results in a single test set.
I believe there is an error in {{NonConcurrentRunListener.describesNewTestSet}}.
{code:java}
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals(
currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals(
currentTestSetDescription.getChildren() );
}
return false;
}
return true;
}
{code}
The value of {{description.getChildren().equals(
currentTestSetDescription.getChildren() );}} should be negated so so different
test suits are properly recognized as test sets.
Fixing this locally works and can be reproduced by:
1. Negating the line and building sure fire locally
2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on
[https://github.com/mpkorstanje/surefire-test]
It does however does result in another unwanted result: A feature file with two
scenarios is now reported as having 3 results. One for each scenario and one
for the feature.
{noformat}
-------------------------------------------------------------------------------
Test set: Some Feature 1
-------------------------------------------------------------------------------
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in
Some_Feature_1
{noformat}
> surefire-junit47 reports results against incorrect test
> -------------------------------------------------------
>
> Key: SUREFIRE-1691
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1691
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support
> Affects Versions: 3.0.0-M3
> Reporter: M.P. Korstanje
> Priority: Major
>
> This is a follow up on:
> - https://issues.apache.org/jira/browse/SUREFIRE-1156
> - [https://github.com/cucumber/cucumber-jvm/issues/865]
> The latest reproducer can be found here:
> -
> [https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
> In short Cucumber has a hierarchy of tests that looks at follows:
> {noformat}
> Class annotated with @RunWith(Cucumber.class)
> |- Feature File 1
> | |- Scenario 1a
> | |- Scenario 1b
> |- Feature File 2
> | |- Scenario 2a
> | |- Scenario 2b
> {noformat}
> Surefire tries to group output in tests sets, either per class or per test
> suite. As Cucumber currently only emits test started/finished events for
> Scenarios sure fire can not group scenarios into a test set. They are instead
> grouped with the previous test set. However even after adding test
> started/finished events for features, sure fire groups all results in a
> single test set.
> I believe this is caused by an error in
> {{NonConcurrentRunListener.describesNewTestSet}}.
> {code:java}
> private boolean describesNewTestSet( Description description )
> {
> if ( currentTestSetDescription != null )
> {
> if ( null != description.getTestClass() )
> {
> return !description.getTestClass().equals(
> currentTestSetDescription.getTestClass() );
> }
> else if ( description.isSuite() )
> {
> return description.getChildren().equals(
> currentTestSetDescription.getChildren() );
> }
> return false;
> }
> return true;
> }
> {code}
> The value of {{description.getChildren().equals(
> currentTestSetDescription.getChildren() );}} should be negated so so
> different test suits are properly recognized as test sets.
> Fixing this locally works and can be reproduced by:
> 1. Negating the line and building sure fire locally
> 2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn
> clean install -DskipTests`
> 3. Running `mvn clean test -Pjunit47` on
> [https://github.com/mpkorstanje/surefire-test]
> It does however does result in another unwanted result: A feature file with
> two scenarios is now reported as having 3 results. One for each scenario and
> one for the feature.
> {noformat}
> -------------------------------------------------------------------------------
> Test set: Some Feature 1
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in
> Some_Feature_1
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
