[jira] [Created] (SUREFIRE-1727) Failures during test template creation are ignored

[Jira]

Thomas Weißschuh created SUREFIRE-1727:
------------------------------------------

             Summary: Failures during test template creation are ignored
                 Key: SUREFIRE-1727
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1727
             Project: Maven Surefire
          Issue Type: Bug
          Components: JUnit 5.x support
    Affects Versions: 3.0.0-M4
            Reporter: Thomas Weißschuh


Junit5 allows dynamic creation of tests via {{ @TestTemplate }} and 
{{TestTemplateInvocationContextProvider}}.
If the {{TestTemplateInvocationContextProvider}} fails with an exception itself 
or Junit itself rejects it then surefire still marks the test as successful.

Both Intellij IDEA and the junit 5 console launcher mark the test method as 
failed.

Examples:
{code:java}
package surefirebug;

import java.util.stream.Stream;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;

public class FailingTestTemplateInvocationContextProvider
    implements TestTemplateInvocationContextProvider {

  @Override
  public boolean supportsTestTemplate(ExtensionContext context) {
    return true;
  }

  @Override
  public Stream<TestTemplateInvocationContext> 
provideTestTemplateInvocationContexts(
      ExtensionContext context) {
    //throw new IllegalStateException(""); // throw a custom exception
    return Stream.of(); // this will be rejected by Junit itself with a 
PreconditionFailedException
  }
}
{code}
{code:java}
package surefirebug;

import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;

@ExtendWith(FailingTestTemplateInvocationContextProvider.class)
class DemonstrationTest {
  @TestTemplate
  void testOne() {
    System.out.println("testOne");
  }
{code}
Display in surefire:
{noformat}
...
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ amps-utils ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running surefirebug.DemonstrationTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s 
- in surefirebug.DemonstrationTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
{noformat}
Display in the console runner:
{noformat}
╷
├─ JUnit Jupiter ✔
│  └─ DemonstrationTest ✔
│     └─ testOne() ✘ No supporting TestTemplateInvocationContextProvider 
provided an invocation context
└─ JUnit Vintage ✔

Failures (1):
  JUnit Jupiter:DemonstrationTest:testOne()
    MethodSource [className = 'surefirebug.DemonstrationTest', methodName = 
'testOne', methodParameterTypes = '']
    => org.junit.platform.commons.PreconditionViolationException: No supporting 
TestTemplateInvocationContextProvider provided an invocation context
       
org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:296)
       
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(TestTemplateTestDescriptor.java:142)
       
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:108)
       
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:41)
       
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
       
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
       
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
       org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
       
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
       
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
       [...]

Test run finished after 40 ms
[         4 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         3 containers successful ]
[         1 containers failed     ]
[         0 tests found           ]
[         0 tests skipped         ]
[         0 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         0 tests failed          ]
{noformat}
I look at the code and this is how the error is communicated to surefire by 
junit.
 {{RunListenerAdapter.executionFinished()}} receives the following data:
{noformat}
testIdentifier.getSource() instanceof MethodSource
testExecutionResult.getStatus() == failed
testExecutionResult.getThrowable() instanceof PreconditionViolationException
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)