GitHub user jontodd opened a pull request:
https://github.com/apache/maven-surefire/pull/47
Support for TestNG successPercentage
TestNG has the concept of "successPercentage" where a probabalistic test
can be run some number of innvoations and then if the successPercentage is met,
the test is considered passing.
Example code might look like:
```java
@Test(invocationCount = 100, successPercentage = 25)
void testRandom() {
// ~50% chance of generating 0 & 1
int num = new Random().nextInt(2);
Assert.assertEquals(num, 1);
}
```
As detailed in SUREFIRE-654, the problem is that surefire is currently
mapping each invocation to a new test run, and any failure, regardless of
whether we're within the allowed success percentage or not will result in this
test failing. This renders any other value for successPercentage aside from 100
useless.
Digging into the code it looks like this was just an oversight in the
inital implementation of the TestNGReporter. The TestNG
ITestListener#onTestFailedButWithinSuccessPercentage method API stipulates that
it will only be called when we're still within the passing limit:
Invoked each time a method fails but has been annotated with
successPercentage and this failure still keeps it within the success percentage
requested.
This change maps tests which pass within the successPercentage as passing
which is safe because if we go beyond the passing threshold the test will be
marked failed as desired.
__What about tests?__
Yes, there are no test changes. It looks like the providers aren't being
tested. I'd be happy to add coverage that asserts the correct stubbed reporter
method is called but that seems like overkill given the existing testing
conventions.
__What about breaking existing users?__
As mentioned earlier, essentially successPercentage wasn't even usable
within surefire before, so this should only provide net-new functionality to
users. In the example above, before my change that test would fail every time
and would result in ~50 failures. After this change the above example would
pass so long as there are only 25 or fewer failures.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jontodd/maven-surefire
testng_success_pct_support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/maven-surefire/pull/47.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #47
----
commit a70bbb8f043f88ccdc1ec7db6810b813688562c2
Author: Jon Todd <[email protected]>
Date: 2014-09-03T00:29:08Z
Support for TestNG successPercentage
TestNG has the concept of "successPercentage" where a probabalistic test
can be run some number of innvoations and then if the successPercentage is met,
the test is considered passing.
Example code might look like:
```java
@Test(invocationCount = 100, successPercentage = 25)
void testRandom() {
// ~50% chance of generating 0 & 1
int num = new Random().nextInt(2);
Assert.assertEquals(num, 1);
}
```
As detailed in SUREFIRE-654, the problem is that surefire is currently
mapping each invocation to a new test run, and any failure, regardless of
whether we're within the allowed success percentage or not will result in this
test failing. This renders any other value for successPercentage aside from 100
useless.
Digging into the code it looks like this was just an oversight in the
inital implementation of the TestNGReporter. The TestNG
ITestListener#onTestFailedButWithinSuccessPercentage method API stipulates that
it will only be called when we're still within the passing limit:
Invoked each time a method fails but has been annotated with
successPercentage and this failure still keeps it within the success percentage
requested.
This change maps tests which pass within the successPercentage as passing
which is safe because if we go beyond the passing threshold the test will be
marked failed once as desired.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
