On 23 Feb 07, at 10:08 PM 23 Feb 07, Brett Porter wrote:
On 04/01/2007, at 4:32 PM, Brett Porter wrote:
Jason - any further thoughts on this?
Ping... No is a valid answer :)
I'd like to get your summary put somewhere individuals can pick
things off to work on - probably a jira project for shared. WDYT?
I put the summary here for now, it's pretty elaborate and can most
likely be worked on easily. Given one or two people have expressed
interested or worked on the ITs I think that will suffice.
http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk/
ITProblems.txt
The top things that could be done:
1) The issues that would be most helpful that could be tackled on a
piecewise basis by many would be to take plugin specific ITs out of
the ITs. There are many in there for the surefire plugin so while
you're doing that you can look at it. Piecewise but probably totally
simple because you have to replace it with an IT that actually tests
what it was testing. A lot of time I have had to make a new IT plugin
flavour.
2) The next issue of importance would be to collect all the in IT
plugin plugins, invokers and verifiers and align all theses.
3) Once 2) is done then we wire the embedder option into the
resulting invoker.
As far as them working in situ: the ITs could now run from the top
level of http://svn.apache.org/repos/asf/maven/core-integration-
testing/trunk/ with an addition to the POM. It's not there because I
generally build from the top-level and then walk into http://
svn.apache.org/repos/asf/maven/core-integration-testing/trunk/core-
integration-tests/. The invoker plugin has been released and that's
what I used to install the supporting artifacts because they can't
install in a reactor run normally because there are 5 artifacts with
the id.
I am not entirely happy with the structure we have but I don't think
it's worth changing right now until 1) and 2) are done. It's not
obvious how to run them from and IDE and that's where I've found it
to be most convenient to run them.
I'm overcommitted for working on things right now, but I know a
couple of people are confused about the IT testing, and we've got
all those chronically broken plugins. Volunteers?
On 18/12/2006, at 4:35 PM, Brett Porter wrote:
On 12/12/2006, at 5:08 AM, Jason van Zyl wrote:
So, in response to John's email: I think we need to settle on
what we're going to use and stop writing new tools. We have
several invokers, several verifiers, several IT plugins, and the
plugin harness. It is simply out of control. We have different
methods being used in different plugins, nothing is standard and
it's going to kill us. The most prevalent tool, as defective as
it might be is what is being used in the ITs themselves.
Stephane managed to use this successfully in some of his
plugins. Then after that we have an array of usages. What should
happen before we start writing more stuff is to figure out
something we can use now, and how to merge what we have together
instead of writing more tools.
Agreed. I think that's what John was getting at too, but by doing
it clean rather than rewriting something that was in use
somewhere. So the work left to do, in either case, is apply it
consistently and get rid of the stuff we aren't going to use.
To me, it looks like:
- the plugin-testing-harness needs to go. They should be
integration tests that use a proper pom, or use pure mocks rather
than the stubs that tend to just have a bunch of impossible-to-
get-under-real-condition values.
- John's test tools have the most complete invocation options,
and tools for managing repositories that we can reuse, so I'd opt
for that in that area
- the verifier is well utilised, so if that is merged with the
code from the verifier plugin then we can lose the invocation
stuff and repository management stuff and merge it all together
Can we have a wiki page with this work list? Or, can we check in
the omni outliner files + an export for the non-mac users to review?
[snip points I agree with]
- [+] The ITs should be in a project of their own so that we
can
reuse them across versions of Maven. We could actually
run
new versions of integration tests against old versions of
Maven. Solution: the ITs are now in a separate build
and it
is possible to run them
How should this play out in plugins? I would be in favour of
separate projects for these.
Two things that are must haves for me:
- integration tests / anything that forks a Maven instance must
*not* be part of the normal test build for a plugin. They take
way too long, and lead to use of maven.test.skip :(
- as far as integration testing in general, I think we recommend
a separate project, but enable them to be part of the same
project for simplicity of development (this is more specific to
things like web applications where it is probably beneficial).
- [ ] We should be able to easily integrate the IT into a
larger
run where we can use forked or embedded execution.
Not sure what you mean here. What is an IT that *doesn't* use
forked or embedded execution?
- [ ] automate the testing of ITs submitted by users
what does this mean? I think, like a patch, a submitted IT still
needs to be reviewed, and incorporated into the main test suite
(with corresponding fix-for version so it only runs when it is
expected to work). Otherwise, long term, we'll have lots of
duplicated or poorly conceived ITs. I've seen test cases
submitted that are quite useful at demonstrating something, but
contain half of the user's proect which is not good for our long
term scalability.
- [ ] Each IT should have its own repository if it needs
resources
from repository. We can't mess with a users repository
when
testing.
- [ ] We need to have a file system based remote repository for
testing
Agreed. Isn't that what John's tool already does?
- [ ] We need to standardize on integration testing in
general. We
have people going all over the place and it's a disaster.
- [ ] We have too many IT plugins (3)
- [ ] We have too many invokers (5)
- [ ] We have too many verifiers (3)
Let's specifically get these mapped out and a path forward so
that everyone can push towards it, rather than relying on you to
do the work.
- [ ] The ITs should run nicely from an IDE. Solution: this
does
work but requires that you run mvn clean
resources:testResources first as the IDE doesn't know
how to
set that up. Needs to be fully fixed. But it is much
nicer
running this stuff in your IDE.
Agree with the point, but not sure what you are referring to
about testResources - the generated projects for IDEA and I think
Eclipse already do this (and obviously better integration will
bring it).
I think the dangerous thing is using resources for non-classpath
resources. It's better for the tests to setup and use a clean
project instance for an IT itself (using helper tools).
[snip specific notes on old ITs that need to be updated]
- [ ] artifactIds should be aligned with directories
Agreed. Also, as I did in the last test I wrote, what do we think
about using purposeful names rather than numbers for integration
tests?
Thanks for this.
Cheers,
Brett
--------------------------------------------------------------------
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
