-- Alexander Kriegisch http://scrum-master.de
Am 28.05.2014 um 05:13 schrieb Benson Margulies <[email protected]>: > On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <[email protected]>wrote: > >> I am an intern at Google. The first step of my internship project is to add >> the ability to Maven to automatically rerun failing tests a few times, to >> see if they ever pass in any of the reruns. It is useful because in many >> cases a test fails because it is flaky, not because there is a bug in the >> new source code change. > > I entirely disagree. 99.9999% of programs are completely deterministic. If > you feed them the same inputs, you get the same output. Well, in my previous project (I am an Agile Coach) my teams were working on a distributed, multi-threaded system and around 10% of all unit tests and next to all integration tests were affected by the exception you mentioned: > The exceptions are tests that use threads, and it's quite hard to do a good > job writing them. One can argue that several of the unit tests were not real unit tests because they mocked some, but not all of the infrastructure they needed for running. Anyway, they were running in Surefire (the integration tests in Failsafe), and some lf them were flaky. > Just 'rerunning a few times' does not lead to useful > information in my view. Only the information that indeed they are flaky and a rough percentage of cases in which they fail, which would be rather dependend on the test environment though. Better than nothing information-wise. So far, so good. Having said all that, I still agree to Benson that a failing test should just fail, period. I do not want developers to think, "it is only failing only in 5% of all runs, so it is fine / low priority to fix". OTOH I understand Qingzhou from a perspective of wishing to "test the test" or rather wishing to harden a flaky test into a stable one, either by fixing the test or the threading bug(s) it exposes. Benson's suggestion below looks promising though, so thanks for that. > if you want random testing, there's a fine alternative at > http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine > maven plugin. It can already 'rerun your tests a few times' very easily. > > I wonder what the more involved surefire maintainers think, but as a PMC > member I'd be -1 on this proposal. > >> We think the right way to achieve this is to modify surefire plugin of >> maven. We want to add it as a part of the configuration of surefire, so >> users can decide whether they want to enable this feature, and how many >> times they want to rerun failing tests. We plan to open-source our >> contribution, and hopefully can merge our code into surefire master branch >> in the end. Therefore, we are wondering do you have any >> comments/suggestions/opinions regarding this? We appreciate any input. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
