Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-02-01 Thread Stephen Connolly
https://git-wip-us.apache.org/repos/asf?p=maven-integration-testing.git;a=commit;h=92a11a96 (not using annotations because our integration tests are JUnit 3.8.x style) On 1 February 2017 at 20:03, Christian Schulte wrote: > Am 02/01/17 um 20:05 schrieb Stephen Connolly: > > I would rather modif

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-02-01 Thread Christian Schulte
Am 02/01/17 um 20:05 schrieb Stephen Connolly: > I would rather modify verifier to allow recording versions we expect to > fail vs versions we expect to pass. +1 On method level instead of class level. Maybe by employing annotations one can add to methods like @SupportedBy.

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-02-01 Thread Michael Osipov
Am 2017-02-01 um 20:05 schrieb Stephen Connolly: I would rather modify verifier to allow recording versions we expect to fail vs versions we expect to pass. I'll see if I can code it up. That would at least give us the verification of the failure on the broken and the verification of the fix wit

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-02-01 Thread Stephen Connolly
I would rather modify verifier to allow recording versions we expect to fail vs versions we expect to pass. I'll see if I can code it up. That would at least give us the verification of the failure on the broken and the verification of the fix without the duplication and consequent risk that users

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-02-01 Thread Michael Osipov
Am 2017-01-31 um 22:25 schrieb Stephen Connolly: Ok so, we'll need to knock this one out and see if there is a consensus. My position is that I only have a slight preference for A over B and I cannot fully articulate why. Michael, do you feel you can present a reasoned argument in favour of A a

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Christian Schulte
Am 01/31/17 um 23:23 schrieb Christian Schulte: > Before the reset I did A. On the branch I did B. Do not ask me why I did > it differently this time. Maybe because I reviewed the versions in more > detail this time. While at it: I somehow get the feeling that those ITs > really should be unit test

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Christian Schulte
Before the reset I did A. On the branch I did B. Do not ask me why I did it differently this time. Maybe because I reviewed the versions in more detail this time. While at it: I somehow get the feeling that those ITs really should be unit tests. I added the exact same tests to the core as unit test

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Stephen Connolly
Ok so, we'll need to knock this one out and see if there is a consensus. My position is that I only have a slight preference for A over B and I cannot fully articulate why. Michael, do you feel you can present a reasoned argument in favour of A and we'll let one of the B proponents present their

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Michael Osipov
Am 2017-01-31 um 20:23 schrieb Stephen Connolly: Looking like a consensus on B. I am actually in favor of A. How do you want to assure with B that the will be properly handled for current master as you fixed the test for released versions? Michael On Tue 31 Jan 2017 at 12:51, Anders Hamma

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Stephen Connolly
Looking like a consensus on B. Let's lazy this one. On Tue 31 Jan 2017 at 12:51, Anders Hammar wrote: > I favor B. > > /Anders > > On Tue, Jan 31, 2017 at 12:42 PM, Stephen Connolly < > stephen.alan.conno...@gmail.com> wrote: > > > We have kind of established a consensus on how to handle the ca

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Anders Hammar
I favor B. /Anders On Tue, Jan 31, 2017 at 12:42 PM, Stephen Connolly < stephen.alan.conno...@gmail.com> wrote: > We have kind of established a consensus on how to handle the case where we > want to change the specification of how Maven works going forward. > Specifically, if we decide that the

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Arnaud Héritier
I would prefer B On Tue, Jan 31, 2017 at 1:34 PM, Igor Fedorenko wrote: > > B. Fix the test, but exclude the broken versions of Maven from the range > with a comment explaining why > > I sometimes rerun integration tests against released versions of Maven > to validate the tests are still workin

Re: [DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Igor Fedorenko
> B. Fix the test, but exclude the broken versions of Maven from the range with a comment explaining why I sometimes rerun integration tests against released versions of Maven to validate the tests are still working and I know other developers who do that too. Having failures would just mean tests

[DISCUSS] How do we want to handle false positives in the integration tests

2017-01-31 Thread Stephen Connolly
We have kind of established a consensus on how to handle the case where we want to change the specification of how Maven works going forward. Specifically, if we decide that the old behaviour of Maven is no longer going to be the new behaviour of Maven our procedure in the integration tests is as f