On Tue, Apr 16, 2019 at 10:07 PM Sergei Nikulov <sergey.niku...@gmail.com> wrote:
> Hello All, > > Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with > --repeat-until-fail <n> option? > > My expectation, if I requesting a test run for example 10 times, the > sequence should be as follows: > > fixture_setup, then test, then fixture_cleanup repeats 10 times > > But in practice, I've got following sequence > > 1. fixture_setup runs 10 times > 2. test runs 10 times > 3. fixture_cleanup runs 10 times > > Is it expected behavior? > I'm using ctest version 3.14.2 > The documentation for the --repeat-until-fail option says "Require each test to run <n> times without failing in order to pass". A fixture setup or cleanup test is still a test, so they will also be run <n> times. The logic that implements repeating tests sets up a counter on each test and it runs that test <n> times before marking that test as complete. This is why you see fixture_setup run 10 times, then test runs 10 times and lastly fixture_cleanup runs 10 times. Whenever I've wanted to use --repeat-until-fail (admittedly not all that common for me), I'm generally only interested in repeatedly running one particular test. For that case, I would typically run the test once with fixture cleanups disabled (i.e. use a command line "ctest -FC . -R myFlakeyTest"). I can then run ctest again for just my flaky test with all fixtures disabled (e.g. "ctest -FA . -R myFlakeyTest --repeat-until-fail 10"). This strategy won't work for everyone's situation, but it has been effective for me. -- Craig Scott Melbourne, Australia https://crascit.com Get the hand-book for every CMake user: Professional CMake: A Practical Guide <https://crascit.com/professional-cmake/>
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
