[
https://issues.apache.org/jira/browse/SUREFIRE-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weißschuh updated SUREFIRE-1727:
---------------------------------------
Description:
Junit5 allows dynamic creation of tests via {{ @TestTemplate }} and
{{TestTemplateInvocationContextProvider}}.
If the {{TestTemplateInvocationContextProvider}} fails with an exception itself
or Junit itself rejects it then surefire still marks the test as successful.
Both Intellij IDEA and the junit 5 console launcher mark the test method as
failed.
Examples:
{code:java}
package surefirebug;
import java.util.stream.Stream;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
public class FailingTestTemplateInvocationContextProvider
implements TestTemplateInvocationContextProvider {
@Override
public boolean supportsTestTemplate(ExtensionContext context) {
return true;
}
@Override
public Stream<TestTemplateInvocationContext>
provideTestTemplateInvocationContexts(
ExtensionContext context) {
//throw new IllegalStateException(""); // throw a custom exception
return Stream.of(); // this will be rejected by Junit itself with a
PreconditionViolationException
}
}
{code}
{code:java}
package surefirebug;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
@ExtendWith(FailingTestTemplateInvocationContextProvider.class)
class DemonstrationTest {
@TestTemplate
void testOne() {
System.out.println("testOne");
}
}
{code}
Display in surefire:
{noformat}
...
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ amps-utils ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running surefirebug.DemonstrationTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s
- in surefirebug.DemonstrationTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
{noformat}
Display in the console runner:
{noformat}
╷
├─ JUnit Jupiter ✔
│ └─ DemonstrationTest ✔
│ └─ testOne() ✘ No supporting TestTemplateInvocationContextProvider
provided an invocation context
└─ JUnit Vintage ✔
Failures (1):
JUnit Jupiter:DemonstrationTest:testOne()
MethodSource [className = 'surefirebug.DemonstrationTest', methodName =
'testOne', methodParameterTypes = '']
=> org.junit.platform.commons.PreconditionViolationException: No supporting
TestTemplateInvocationContextProvider provided an invocation context
org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:296)
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(TestTemplateTestDescriptor.java:142)
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:108)
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:41)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[...]
Test run finished after 40 ms
[ 4 containers found ]
[ 0 containers skipped ]
[ 4 containers started ]
[ 0 containers aborted ]
[ 3 containers successful ]
[ 1 containers failed ]
[ 0 tests found ]
[ 0 tests skipped ]
[ 0 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 0 tests failed ]
{noformat}
I look at the code and this is how the error is communicated to surefire by
junit.
{{RunListenerAdapter.executionFinished()}} receives the following data:
{noformat}
testIdentifier.getSource() instanceof MethodSource
testExecutionResult.getStatus() == failed
testExecutionResult.getThrowable() instanceof PreconditionViolationException
{noformat}
was:
Junit5 allows dynamic creation of tests via {{ @TestTemplate }} and
{{TestTemplateInvocationContextProvider}}.
If the {{TestTemplateInvocationContextProvider}} fails with an exception itself
or Junit itself rejects it then surefire still marks the test as successful.
Both Intellij IDEA and the junit 5 console launcher mark the test method as
failed.
Examples:
{code:java}
package surefirebug;
import java.util.stream.Stream;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
public class FailingTestTemplateInvocationContextProvider
implements TestTemplateInvocationContextProvider {
@Override
public boolean supportsTestTemplate(ExtensionContext context) {
return true;
}
@Override
public Stream<TestTemplateInvocationContext>
provideTestTemplateInvocationContexts(
ExtensionContext context) {
//throw new IllegalStateException(""); // throw a custom exception
return Stream.of(); // this will be rejected by Junit itself with a
PreconditionFailedException
}
}
{code}
{code:java}
package surefirebug;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
@ExtendWith(FailingTestTemplateInvocationContextProvider.class)
class DemonstrationTest {
@TestTemplate
void testOne() {
System.out.println("testOne");
}
{code}
Display in surefire:
{noformat}
...
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ amps-utils ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running surefirebug.DemonstrationTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s
- in surefirebug.DemonstrationTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
{noformat}
Display in the console runner:
{noformat}
╷
├─ JUnit Jupiter ✔
│ └─ DemonstrationTest ✔
│ └─ testOne() ✘ No supporting TestTemplateInvocationContextProvider
provided an invocation context
└─ JUnit Vintage ✔
Failures (1):
JUnit Jupiter:DemonstrationTest:testOne()
MethodSource [className = 'surefirebug.DemonstrationTest', methodName =
'testOne', methodParameterTypes = '']
=> org.junit.platform.commons.PreconditionViolationException: No supporting
TestTemplateInvocationContextProvider provided an invocation context
org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:296)
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(TestTemplateTestDescriptor.java:142)
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:108)
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:41)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[...]
Test run finished after 40 ms
[ 4 containers found ]
[ 0 containers skipped ]
[ 4 containers started ]
[ 0 containers aborted ]
[ 3 containers successful ]
[ 1 containers failed ]
[ 0 tests found ]
[ 0 tests skipped ]
[ 0 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 0 tests failed ]
{noformat}
I look at the code and this is how the error is communicated to surefire by
junit.
{{RunListenerAdapter.executionFinished()}} receives the following data:
{noformat}
testIdentifier.getSource() instanceof MethodSource
testExecutionResult.getStatus() == failed
testExecutionResult.getThrowable() instanceof PreconditionViolationException
{noformat}
> Failures during test template creation are ignored
> --------------------------------------------------
>
> Key: SUREFIRE-1727
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1727
> Project: Maven Surefire
> Issue Type: Bug
> Components: JUnit 5.x support
> Affects Versions: 3.0.0-M4
> Reporter: Thomas Weißschuh
> Priority: Major
>
> Junit5 allows dynamic creation of tests via {{ @TestTemplate }} and
> {{TestTemplateInvocationContextProvider}}.
> If the {{TestTemplateInvocationContextProvider}} fails with an exception
> itself or Junit itself rejects it then surefire still marks the test as
> successful.
> Both Intellij IDEA and the junit 5 console launcher mark the test method as
> failed.
> Examples:
> {code:java}
> package surefirebug;
> import java.util.stream.Stream;
> import org.junit.jupiter.api.extension.ExtensionContext;
> import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
> import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
> public class FailingTestTemplateInvocationContextProvider
> implements TestTemplateInvocationContextProvider {
> @Override
> public boolean supportsTestTemplate(ExtensionContext context) {
> return true;
> }
> @Override
> public Stream<TestTemplateInvocationContext>
> provideTestTemplateInvocationContexts(
> ExtensionContext context) {
> //throw new IllegalStateException(""); // throw a custom exception
> return Stream.of(); // this will be rejected by Junit itself with a
> PreconditionViolationException
> }
> }
> {code}
> {code:java}
> package surefirebug;
> import org.junit.jupiter.api.TestTemplate;
> import org.junit.jupiter.api.extension.ExtendWith;
> @ExtendWith(FailingTestTemplateInvocationContextProvider.class)
> class DemonstrationTest {
> @TestTemplate
> void testOne() {
> System.out.println("testOne");
> }
> }
> {code}
> Display in surefire:
> {noformat}
> ...
> [INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ amps-utils ---
> [INFO]
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running surefirebug.DemonstrationTest
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009
> s - in surefirebug.DemonstrationTest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> {noformat}
> Display in the console runner:
> {noformat}
> ╷
> ├─ JUnit Jupiter ✔
> │ └─ DemonstrationTest ✔
> │ └─ testOne() ✘ No supporting TestTemplateInvocationContextProvider
> provided an invocation context
> └─ JUnit Vintage ✔
> Failures (1):
> JUnit Jupiter:DemonstrationTest:testOne()
> MethodSource [className = 'surefirebug.DemonstrationTest', methodName =
> 'testOne', methodParameterTypes = '']
> => org.junit.platform.commons.PreconditionViolationException: No
> supporting TestTemplateInvocationContextProvider provided an invocation
> context
>
> org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:296)
>
> org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(TestTemplateTestDescriptor.java:142)
>
> org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:108)
>
> org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:41)
>
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
>
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
>
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
>
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
>
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> [...]
> Test run finished after 40 ms
> [ 4 containers found ]
> [ 0 containers skipped ]
> [ 4 containers started ]
> [ 0 containers aborted ]
> [ 3 containers successful ]
> [ 1 containers failed ]
> [ 0 tests found ]
> [ 0 tests skipped ]
> [ 0 tests started ]
> [ 0 tests aborted ]
> [ 0 tests successful ]
> [ 0 tests failed ]
> {noformat}
> I look at the code and this is how the error is communicated to surefire by
> junit.
> {{RunListenerAdapter.executionFinished()}} receives the following data:
> {noformat}
> testIdentifier.getSource() instanceof MethodSource
> testExecutionResult.getStatus() == failed
> testExecutionResult.getThrowable() instanceof PreconditionViolationException
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
