[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815514#comment-17815514
]
Geoff Soutter commented on SUREFIRE-2147:
-----------------------------------------
This would help with making DynamicTests more usable in IDEs.
https://github.com/junit-team/junit5/issues/3682
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---------------------------------------------------------------------------
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M7, 3.2.5
> Reporter: Geoff Soutter
> Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection<DynamicNode> createDynamicTests() {
> List<DynamicNode> folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List<DynamicNode> folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List<DynamicNode> rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> **** folder2
> ***** test1
> ***** test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> However, things are not as good when using surefire as it still uses the
> years old ant XSD for XML reporting and that XSD doesn't support anything
> apart from basic Java usage, certainly nothing like dynamic tests which don't
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates XML report content in
> surefire-reports like so
> {code}
> <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
> time="0.012"/>
> <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
> time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
> <testcase name="folder2 folder1 createDynamicTests() test1"
> classname="pkg.JUnit5DynamicTest" time="0.011"/>
> <testcase name="folder2 folder1 createDynamicTests() test2"
> classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it (eg
> Jenkins JUnit plugin).
> * should surefire uses be forced to enable phrased naming so that reporting
> tools can access the dynamic folder and test names?
> * how can XML reporting tools show the nested structure of the
> DynamicFolders?
> ** can XML reporting tools easily tell whether the XML was generated with
> phrased or non phrased settings? Should they need to?
> ** if using phrased, should they parse the phrased name attribute to find the
> dynamic folder and test names?
> ** if using non phrased, is there a way of surefire safely injecting the
> dynamic folder and test names into an existing attribute which will work in a
> backwards compatible way with the XML reporting tools?
> Also the internal ordering of the phrased name attribute seems inconsistent,
> or at least I don't understand it. Since "test" is inside "folder1", surely
> "folder1" and "test" should be next to each other rather than at opposite
> ends of the value? If we expect ordering by heirarchy position then we would
> expect the folder/method names like "createDynamicTest() folder1 folder2
> test1".
> h3. Solution Ideas
> Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
> supported as XML report format and at that point, I guess support for dynamic
> tests would be easy to add. However that could be years off, if at all.
> Anyway, back to the real world, Surefire using the old ant XSD can generate
> two variants of the XML report, using non phrased or phrased names. Being
> able to control the phrased and non phrased attributes individually seems to
> make no sense as the attributes are related. But anyway, we have to make do.
> Seems for JUnit5 XML reporting, the intention for the "name" attribute value
> was:
> * non phrased is the TestIdentifier.legacyReportingName
> * phrased is the TestIdentifier.displayName
> h3. Non Phrased
> Dynamic tests with non phrased method names are currently not usable. So
> anything we do here would be an improvement.
> The JUnit5 legacyReportingName for the example looks like
> "createDynamicTests[1][1][1]". However this is basically no use as it uses
> synthetic ids rather than the real names and will end up as a flat list
> without any hierarchical structure in XML reporting tools. From memory. this
> was actually used in earlier surefire versions, not sure what happened there,
> I guess the fixes for Parameterized tests broke it.
> To do something which solves the hierarchical structure problem, we could
> instead add the dynamic names as synthetic child packages in the classname
> attribute.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
> <testcase name="test1"
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> <testcase name="test2"
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> {code}
> Yes it is ugly, but it should allow us to view all the tests in a
> heirarchical structure inside existing XML reporting tools.
> h3. Phrased
> For phrased usage, its less obvious what to do.
> The nested structure is injected into a single XML attribute and I haven't
> seen any documentation of the internal structure of that attribute.
> The ordering looks inconsistent as it is and possibly needs fixing. However,
> changing this value without any schema as to what the contents mean seems
> dangerous.
> Casual testing with an available Jenkins instance seems to indicate that
> Jenkins JUnit already has some capability to parse the phrased method name as
> it shows folder2 as a "package" (but folder1 is lost).
> To get an overall useful result here I guess we'd need to
> * create and publish some kind of docs / schema on the internal structure we
> want to support for downstream reporting tools
> * adjust whatever output we have now to meet the docs/schema, if required
> * wait for downstream reporting tools to support it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
