[ https://issues.apache.org/jira/browse/SUREFIRE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878350#comment-17878350 ]
M.P. Korstanje edited comment on SUREFIRE-2260 at 8/31/24 3:02 PM: ------------------------------------------------------------------- The problem seems to be a mismatch in assumptions between Surefire and the JUnit Platform. Surefire still assumes that all tests have a class and method name. With the JUnit Platform this is no longer the case. The Cucumber Scenarios are provided to the JUnit Platform as [TestDescriptors|https://junit.org/junit5/docs/current/api/org.junit.platform.engine/org/junit/platform/engine/TestDescriptor.html#getDisplayName()] with identical display names, but unique ids. This can be seen by breaking in {{SuiteEngine.discover}} and inspecting the {{engineDescriptor}}. (There may be a few invocations). {code:java} engineDescriptor.getDescendants().stream().map(Object::toString).collect(Collectors.toList()) // Results in: result = {ArrayList@5282} size = 6 0 = "SuiteTestDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]" 1 = "CucumberEngineDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]" 2 = "FeatureDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]" 3 = "PickleDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]/[scenario:3]" 4 = "PickleDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]/[scenario:6]" 5 = "PickleDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]/[scenario:9]" engineDescriptor.getDescendants().stream().map(TestDescriptor::getDisplayName).collect(Collectors.toList()) // Results in: result = {ArrayList@5153} size = 6 0 = "CucumberReproducerTest" 1 = "Cucumber" 2 = "Reproducer" 3 = "identical name" 4 = "identical name" 5 = "identical name" {code} Then based of their display name the {{RunListenerAdapter.createReportEntry}} then maps these last 3 to a {{SimpleReportEntry}} with {{className}} "reproducer" and {{methodName}} "identical name". The {{StatelessXmlReporter}} keeps track of the executed tests in {{testClassMethodRunHistoryMap}} by their {{className}} and {{methodName}}. So the results of all three scenarios get intermixed here. So when {{serializeTestClassWithRerun}} is invoked it will handle the {{FLAKE}} case and then try to print each result: {code:java} for (WrappedReportEntry singleRunEntry : methodEntries) { if (singleRunEntry.getReportEntryType() != SUCCESS) { getTestProblems( fw, ppw, singleRunEntry, trimStackTrace, outputStream, singleRunEntry.getReportEntryType().getFlakyXmlTag(), true); } } {code} And while {{SUCCESS}} states are excluded here, {{singleRunEntry.getReportEntryType().getFlakyXmlTag()}} can also return the empty string for {{SKIPPED}}. Which is unexpected, because skipped tests are never retried, but we managed to sneak it in here earlier. I reckon the proper solution would be to make sure that {{SimpleReportEntry}} has an explicit field for ids, separate from the method and class names. was (Author: mpkorstanje): The problem seems to be a mismatch in assumptions between Surefire and the JUnit Platform. Surefire still assumes that all tests have a class and method name. With the JUnit Platform this is no longer the case. The Cucumber Scenarios are provided to the JUnit Platform as [TestDescriptors| https://junit.org/junit5/docs/current/api/org.junit.platform.engine/org/junit/platform/engine/TestDescriptor.html#getDisplayName()] with identical display names, but unique ids. This can be seen by breaking in {{SuiteEngine.discover}} and inspecting the {{engineDescriptor}}. (There may be a few invocations). {code:java} engineDescriptor.getDescendants().stream().map(Object::toString).collect(Collectors.toList()) // Results in: result = {ArrayList@5282} size = 6 0 = "SuiteTestDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]" 1 = "CucumberEngineDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]" 2 = "FeatureDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]" 3 = "PickleDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]/[scenario:3]" 4 = "PickleDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]/[scenario:6]" 5 = "PickleDescriptor: [engine:junit-platform-suite]/[suite:io.cucumber.skeleton.CucumberReproducerTest]/[engine:cucumber]/[feature:classpath%3Aio%2Fcucumber%2Fskeleton%2Freproducer.feature]/[scenario:9]" engineDescriptor.getDescendants().stream().map(TestDescriptor::getDisplayName).collect(Collectors.toList()) // Results in: result = {ArrayList@5153} size = 6 0 = "CucumberReproducerTest" 1 = "Cucumber" 2 = "Reproducer" 3 = "identical name" 4 = "identical name" 5 = "identical name" {code} Then based of their display name the {{RunListenerAdapter.createReportEntry}} then maps these last 3 to a {{SimpleReportEntry}} with {{className}} "reproducer" and {{methodName}} "identical name". The {{StatelessXmlReporter}} keeps track of the executed tests in {{testClassMethodRunHistoryMap}} by their {{className}} and {{methodName}}. So the results of all three scenarios get intermixed here. So when {{serializeTestClassWithRerun}} is invoked it will handle the {{FLAKE}} case and then try to print each result: {code:java} for (WrappedReportEntry singleRunEntry : methodEntries) { if (singleRunEntry.getReportEntryType() != SUCCESS) { getTestProblems( fw, ppw, singleRunEntry, trimStackTrace, outputStream, singleRunEntry.getReportEntryType().getFlakyXmlTag(), true); } } {code} And while {{SUCCESS}} states are excluded here, {{singleRunEntry.getReportEntryType().getFlakyXmlTag()}} can also return the empty string for {{SKIPPED}}. Which is unexpected, because skipped tests are never retried, but we managed to sneak it in here earlier. I reckon the proper solution would be to make sure that {{SimpleReportEntry}} has an explicit field for ids, separate from the method and class names. > Element name cannot be empty when rerunning identically named scenarios > ----------------------------------------------------------------------- > > Key: SUREFIRE-2260 > URL: https://issues.apache.org/jira/browse/SUREFIRE-2260 > Project: Maven Surefire > Issue Type: Bug > Reporter: M.P. Korstanje > Priority: Major > > Using Cucumber with the JUnit (5) Platform, executing the following scenarios > with {{rerunFailingTestsCount}} set to 2: > {code:none} > Feature: Reproducer > Scenario: identical name > Given this scenario is aborted > Scenario: identical name > Given this scenario fails > Scenario: identical name > Given this scenario passes > {code} > {code:java} > package io.cucumber.skeleton; > import io.cucumber.java.en.*; > import org.junit.jupiter.api.Assertions; > import org.junit.jupiter.api.Assumptions; > public class StepDefinitions { > @Given("this scenario is aborted") > public void thisScenarioIsSkipped() { > Assumptions.assumeTrue(false); > } > @Given("this scenario fails") > public void thisScenarioFails() { > Assertions.assertTrue(false); > } > @Given("this scenario passes") > public void thisScenarioPasses() { > } > } > {code} > Results in: > {code:none} > # Created at 2024-08-31T16:07:54.994 > ForkStarter IOException: Element name cannot be empty > Element name cannot be empty > Element name cannot be empty > Element name cannot be empty > Element name cannot be empty > Element name cannot be empty. > org.apache.maven.plugin.surefire.booterclient.output.MultipleFailureException: > Element name cannot be empty > Element name cannot be empty > Element name cannot be empty > Element name cannot be empty > Element name cannot be empty > Element name cannot be empty > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.<init>(ThreadedStreamConsumer.java:59) > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer.<init>(ThreadedStreamConsumer.java:107) > at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:546) > at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285) > at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250) > at > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1337) > at > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1137) > at > org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:971) > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137) > at > org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370) > at > org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81) > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:298) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:196) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:569) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348) > Suppressed: java.lang.IllegalArgumentException: Element name cannot be > empty > at > org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:468) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:328) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:236) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:190) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:86) > at > org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59) > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86) > at java.base/java.lang.Thread.run(Thread.java:840) > Suppressed: java.lang.IllegalArgumentException: Element name cannot be > empty > at > org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:468) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:328) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:236) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:190) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:86) > at > org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59) > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86) > at java.base/java.lang.Thread.run(Thread.java:840) > Suppressed: java.lang.IllegalArgumentException: Element name cannot be > empty > at > org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:468) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:328) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:236) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:190) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:86) > at > org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59) > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86) > at java.base/java.lang.Thread.run(Thread.java:840) > Suppressed: java.lang.IllegalArgumentException: Element name cannot be > empty > at > org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:468) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:328) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:236) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:190) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:86) > at > org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59) > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86) > at java.base/java.lang.Thread.run(Thread.java:840) > Suppressed: java.lang.IllegalArgumentException: Element name cannot be > empty > at > org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:468) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:328) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:236) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:190) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:86) > at > org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59) > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86) > at java.base/java.lang.Thread.run(Thread.java:840) > Suppressed: java.lang.IllegalArgumentException: Element name cannot be > empty > at > org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:468) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:328) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:236) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:190) > at > org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:86) > at > org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303) > at > org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59) > at > org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86) > at java.base/java.lang.Thread.run(Thread.java:840) > {code} > Minimal reproducer: > [https://github.com/mpkorstanje/surefire-rerun-non-unique-scenarios] > Originally reported as: https://github.com/cucumber/cucumber-jvm/issues/2911 -- This message was sent by Atlassian Jira (v8.20.10#820010)