Found a bug in maven-failsafe-plugin ( https://github.com/apache/accumulo/issues/411) when using groups (JUnit categories): http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#groups http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#excludedGroups
The problem is when the class corresponding to a configured group is not on the class path. This is particularly problematic in multi-module builds, if a group is specified which exists in one module but not another one. It appears the bug occurs because the configured group/category is loaded from the class path, and then compared to the classes to find matching tests. However, this seems backwards. If the tests were examined first, then the name of the groups could be matched against the configuration without loading to load the configured groups/category. Even if this is not possible due to some limitation on how these groups are passed to JUnit or some other restriction, a ClassNotFoundException (CNFE) should be handled more sanely: For example, in the case of a CNFE for the "groups" parameter, then it's clearly not possible for any tests to be included, so no tests should run. In the case of a CNFE for "excludedGroups", then it's clearly not possible for any tests to match, so none should be excluded. This bug has got me thinking about a related feature I'd like to see in maven-surefire-plugin and maven-failsafe-plugin: I'd like to see groups / excludedGroups specified by patterns, similar to `-Dtest=MyTest` instead of `-Dtest=org.apache.project.package.subpackage.testing.MyTest`, it would be nice to set `-Dgroups=MyGroup` rather than `-Dgroups=org.apache.project.package.subpackage.testing.categories.MyGroup`. If groups could be specified as patterns, it would be more obvious that a CNFE could not occur... if a pattern didn't match any classes, then it wouldn't result in a CNFE... it would simply return an empty set. Thanks, Christopher
