[jira] [Commented] (SUREFIRE-1741) Exceptions in parameterized test sources are ignored

[Albert Johnston (Jira)]

    [ 
https://issues.apache.org/jira/browse/SUREFIRE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17005803#comment-17005803
 ] 

Albert Johnston commented on SUREFIRE-1741:
-------------------------------------------

[~tibordigana] I didn't set anything up with TestExecutionListener, let me know 
if that's necessary, but here's the code above running with just JUnit 5's 
console runner. 

 

[https://github.com/ajohnstonTE/JUnit5Test]

 

This is the output if you comment out everything except the exceptional test 
and its MethodSource, then run it:

 
{noformat}
PS D:\development\JUnit5Test> javac -d out -cp 
junit-platform-console-standalone-1.6.0-M1.jar TestClass.java
>> java -jar junit-platform-console-standalone-1.6.0-M1.jar --class-path out 
>> --scan-class-path                          
Thanks for using JUnit! Support its development at 
https://junit.org/sponsoring.
+-- JUnit Jupiter [OK]
| '-- TestClass [OK]
|   '-- willNotBeLoggedAsFailingDespiteNotRunning() 
[X] Stop
'-- JUnit Vintage [OK]Failures (1):
  JUnit Jupiter:TestClass:willNotBeLoggedAsFailingDespiteNotRunning()
    MethodSource [className = 'TestClass', methodName = 
'willNotBeLoggedAsFailingDespiteNotRunning', methodParameterTypes = '']
    => java.lang.RuntimeException: Stop
       TestClass.throwException(TestClass.java:25)
       java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
       
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       java.base/java.lang.reflect.Method.invoke(Method.java:566)
       [...]
       Suppressed: org.junit.platform.commons.PreconditionViolationException: 
Configuration error: You must configure at least one set of arguments for this 
@ParameterizedTest
         
org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:281)
         
org.junit.jupiter.params.ParameterizedTestExtension.lambda$provideTestTemplateInvocationContexts$6(ParameterizedTestExtension.java:90)
         
java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)
         
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
         [...]Test run finished after 49 ms
[         4 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         3 containers successful ]
[         1 containers failed     ]
[         0 tests found           ]
[         0 tests skipped         ]
[         0 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         0 tests failed          ]{noformat}
Some of the output is a little wonky (this is in Powershell), but it otherwise 
seems to work fine. What's interesting is the fact that it says "1 containers 
failed", but also "0 tests failed". I'll skip posting another large chunk of 
logs, but when you run it with the other tests included, it still says "1 
containers failed", but with the correct (aside from the exceptional test) 
number of tests passing/failing.

It seems like I should, but let me know if I should log this as a bug in JUnit 
instead.

 

> Exceptions in parameterized test sources are ignored
> ----------------------------------------------------
>
>                 Key: SUREFIRE-1741
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1741
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: JUnit 5.x support, Maven Failsafe Plugin, Maven Surefire 
> Plugin
>    Affects Versions: 3.0.0-M4
>            Reporter: Albert Johnston
>            Priority: Major
>
> Associated versions on my end at time of testing:
>  * JUnit Jupiter 5.5.1
>  * JDK 11.0.5
>  * Maven 3.6.1
>  
> If the following code is run, either for Surefire or Failsafe, only two of 
> the test methods are logged as having been detected at all. The third one is 
> completely ignored as if it didn't exist.
> {code:java}
> public class ExampleTest {
>   @Test
>   public void shouldRunAndPassAsExpected() {
>   }
>   public static Stream<Arguments> nothing() {
>     return Stream.of(Arguments.arguments());
>   }
>   @ParameterizedTest
>   @MethodSource("nothing")
>   public void shouldRunAndFailAsExpected() {
>     Assertions.fail();
>   }
>   static Stream<Arguments> throwException() {
>     if (true) throw new RuntimeException("Stop");
>     return Stream.of(Arguments.arguments());
>   }
>   @ParameterizedTest
>   @MethodSource("throwException")
>   void willNotBeLoggedAsFailingDespiteNotRunning() {
>     Assertions.fail();
>   }
> }
> {code}
> This is particularly obvious if the first two test methods are commented out, 
> as running mvn test will log that 0 tests were run.
>  
> In case it helps, here's the output I'm seeing when running it with the first 
> two tests commented out:
> {noformat}
> [INFO] -------------------------------------------------------
> [INFO]  T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running ExampleIT
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 s 
> - in ExampleIT
> [INFO] 
> [INFO] Results:
> [INFO] 
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0{noformat}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)