My first approach would also be to handle this in the test framework, e.g. using the JUnit Rules as Kristian already suggested.
There you can decide on a much more fine-grained level what you want to do in case of a failing test: re-run all the tests of the class? Or only those test cases that failed? Perform the re-run of a test case immediately, or after the other tests in the same class were executed? 2014-05-28 21:43 GMT+02:00 Kristian Rosenvold <[email protected]> : > I would really like to see and understand properly how this would > collaborate with JUnit rules for retries. I suspect there might be > some interesting issues regarding reporting (i.e. can the existing > logic handle reporting of n different executions of a single test). > How does this explode i parallel junit is selected :) ? > > Technically you can just pass a system property from your pom and into > the surefire fork with the number of retries, you can read this > proprety in the junit rule. I really think you should start with this > approach and flesh out the details from there. It might turn out this > is all you need, or that some other entirely unforseen issues pop up. > > As for the actual property in the plugin, I really think we should see > where this leads before deciding on how to do this. On a similar note, > I know some people with large corporate clouds sometimes run a test on > 10000 different nodes in parallel, to investigate if there is any kind > of 1/10000 type failures, just to be able to summarize in 20 seconds > instead of 10000 linear runs. > > This might also be another one of those cases where we could consider > picking up user-written code from the test classpath; but I think I > need another year thinking about that :) > > Kristian > > > > > 2014-05-28 19:58 GMT+02:00 Jason van Zyl <[email protected]>: > > I personally don't see an issue with those feature additions, but > Kristian does the lions share of the work on the Surefire plugin so it > would be his call. I would definitely support the additions of those > features and would help test. > > > > On May 28, 2014, at 1:46 PM, Qingzhou Luo <[email protected]> > wrote: > > > >> Hi Jason, > >> > >> Thanks for your reply. We plan to first extend surefire, so users can > put a boolean parameter rerunFailintTests and an int parameter > rerunFailintTestsCount, to tell surefire whether to rerun failing tests > immediately after they fail. If a test passes in any of those reruns if > will be marked as pass, but all the failing attempts will still be recorded > and may be presented to users in a different way. > >> > >> We plan to finish everything in next 1-2 months. So we wonder would you > be willing to accept our contribution in the end, and patch our code into > surefire? > >> > >> Thanks, > >> > >> Qingzhou > >> > >> > >> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <[email protected]> wrote: > >> > >> On May 27, 2014, at 1:14 PM, Qingzhou Luo <[email protected]> > wrote: > >> > >>> Hi, > >>> > >>> I am an intern at Google. The first step of my internship project is > to add > >>> the ability to Maven to automatically rerun failing tests a few times, > to > >>> see if they ever pass in any of the reruns. It is useful because in > many > >>> cases a test fails because it is flaky, not because there is a bug in > the > >>> new source code change. > >>> > >> > >> I think generally being able to re-run failed tests is useful. I use > this feature in the IDE all the time in a fail/fix/run/repeat scenario. If > you're trying to identify flaky tests I think that's useful especially if > you can collect statistics. While In the long term you may not want to rely > on the re-running of failed tests that later succeed without alteration as > the basis for good testing, but it's a good identification tool, but a > useful feature in its own right. > >> > >>> We think the right way to achieve this is to modify surefire plugin of > >>> maven. We want to add it as a part of the configuration of surefire, so > >>> users can decide whether they want to enable this feature, and how many > >>> times they want to rerun failing tests. We plan to open-source our > >>> contribution, and hopefully can merge our code into surefire master > branch > >>> in the end. Therefore, we are wondering do you have any > >>> comments/suggestions/opinions regarding this? We appreciate any input. > >>> > >> > >> Maybe a simple mechanism where you store the failed tests in a file and > then have a flag to run only the failed tests. I think this would be very > useful. Then possible a small, configurable loop around the main execution > of tests if you wanted to specify how many times to run the tests. > >> > >>> Thank you very much! > >>> > >>> Regards, > >>> > >>> Qingzhou > >> > >> Thanks, > >> > >> Jason > >> > >> ---------------------------------------------------------- > >> Jason van Zyl > >> Founder, Apache Maven > >> http://twitter.com/jvanzyl > >> http://twitter.com/takari_io > >> --------------------------------------------------------- > >> > >> The modern conservative is engaged in one of man's oldest exercises in > moral philosophy; that is, > >> the search for a superior moral justification for selfishness. > >> > >> -- John Kenneth Galbraith > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > > > > Thanks, > > > > Jason > > > > ---------------------------------------------------------- > > Jason van Zyl > > Founder, Apache Maven > > http://twitter.com/jvanzyl > > http://twitter.com/takari_io > > --------------------------------------------------------- > > > > Selfish deeds are the shortest path to self destruction. > > > > -- The Seven Samuari, Akira Kurosawa > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
