[
https://jira.codehaus.org/browse/SUREFIRE-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=355203#comment-355203
]
Andreas Gudian commented on SUREFIRE-779:
-----------------------------------------
JUnit and likewise TestNG do ensure that different tests always have different
names, so we need to rely on the run reporter to tell us that a test was either
successful, failed, errorneous, or skipped... Writing that... perhaps we can
make sure that we increase the counter only on "test starting" and "skipped",
but not on success / failure / error. Perhaps that could help making the
console reporting more stable.
Regarding the XML report (which is consumed by CI servers), that's a different
thing.. I don't think the format currently supports adding multiple error
messages and stacktraces to one test tag. Changing that would certainly break
things for the CI servers, at least you would probably end up seeing only the
first error on web page. I think that would be far worse than having slight
fluctuations in the number of overall tests.
> Surefire reports wrong number of failed tests when using JUnit's
> ErrorCollector rule
> ------------------------------------------------------------------------------------
>
> Key: SUREFIRE-779
> URL: https://jira.codehaus.org/browse/SUREFIRE-779
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.10
> Reporter: Lieven Doclo
> Assignee: Tibor Digana
>
> When running a test that contains an ErrorCollector, test failures are not
> correct when an errorcollector contains failures. For example:
> {noformat}
> public class ExampleTest {
> @Rule
> public ErrorCollector errorCollector = new ErrorCollector();
> @Test
> public void testWithErrorCollector() {
> errorCollector.checkThat(true, is(false));
> errorCollector.checkThat(3, is(4));
> }
> }
> {noformat}
> Surefire will reports 2 tests run (of which all failed). However, when you
> fix the same test so that no checks fail, Surefire will report 1 test run.
> This is incorrect, as this will result in false information in systems like
> Hudson or Sonar, the number of testcase should not fluctuate. Surefire should
> only report 1 test run, as that reflects the actual situation.
--
This message was sent by Atlassian JIRA
(v6.1.6#6162)
