paplorinc commented on a change in pull request #333:
URL: https://github.com/apache/maven-surefire/pull/333#discussion_r566785137



##########
File path: 
maven-failsafe-plugin/src/main/java/org/apache/maven/plugin/failsafe/VerifyMojo.java
##########
@@ -139,6 +139,13 @@
     @Parameter( property = "failIfNoTests" )
     private Boolean failIfNoTests;
 
+    /**
+     * Set this to a value greater than 0 to fail the whole test set if the 
cumulative number of flakes reaches
+     * this threshold. Set to 0 to allow an unlimited number of flakes.

Review comment:
       based on the code a negative number would do the same, right?
   Should we maybe rather fail in that case?

##########
File path: 
maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/SurefireHelper.java
##########
@@ -142,7 +142,10 @@ public static void reportExecution( 
SurefireReportParameters reportParameters, R
                                         PluginConsoleLogger log, Exception 
firstForkException )
         throws MojoFailureException, MojoExecutionException
     {
-        if ( firstForkException == null && !result.isTimeout() && 
result.isErrorFree() )
+        boolean isError = firstForkException != null || result.isTimeout() || 
!result.isErrorFree();
+        boolean isFlaky = reportParameters.getFailOnFlakeCount() > 0
+            && result.getFlakes() >= reportParameters.getFailOnFlakeCount();

Review comment:
       minor: could we have `getFailOnFlakeCount` on the same side on both 
cases to simplify comparison?

##########
File path: 
maven-surefire-common/src/test/java/org/apache/maven/plugin/surefire/SurefireHelperTest.java
##########
@@ -156,4 +156,59 @@ public void shouldHandleTestFailure() throws Exception
                 + "for the individual test results.\nPlease refer to dump 
files (if any exist) "
                 + "[date].dump, [date]-jvmRun[N].dump and [date].dumpstream.'" 
);
     }
+
+    @Test
+    public void failsIfThereAreTooManyFlakes() throws Exception
+    {
+        RunResult summary = new RunResult( 1, 0, 0, 0, 1 );
+        Mojo reportParameters = new Mojo();
+        reportParameters.setFailOnFlakeCount( 1 );
+        try
+        {
+            reportExecution( reportParameters, summary, null, null );
+        }
+        catch ( MojoFailureException e )
+        {
+            assertThat( e.getLocalizedMessage() )
+                .isEqualTo( "There are too many flakes.\n\nPlease refer to 
null "
+                    + "for the individual test results.\nPlease refer to dump 
files (if any exist) "
+                    + "[date].dump, [date]-jvmRun[N].dump and 
[date].dumpstream." );
+            return;
+        }
+        fail( "Expected MojoFailureException with message "

Review comment:
       minor: you could add this inside the try in which case we wouldn't need 
a return in the catch

##########
File path: 
maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/SurefireHelper.java
##########
@@ -142,7 +142,10 @@ public static void reportExecution( 
SurefireReportParameters reportParameters, R
                                         PluginConsoleLogger log, Exception 
firstForkException )
         throws MojoFailureException, MojoExecutionException
     {
-        if ( firstForkException == null && !result.isTimeout() && 
result.isErrorFree() )
+        boolean isError = firstForkException != null || result.isTimeout() || 
!result.isErrorFree();
+        boolean isFlaky = reportParameters.getFailOnFlakeCount() > 0

Review comment:
       this stores more than whether we're flaky, rather something like: 
`isFlakyError`, right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to