On Tue, Apr 16, 2019 at 11:36 PM Sergei Nikulov <sergey.niku...@gmail.com> wrote:
> > > > On Tue, Apr 16, 2019 at 10:07 PM Sergei Nikulov < > sergey.niku...@gmail.com> wrote: > >> > >> Hello All, > >> > >> Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with > >> --repeat-until-fail <n> option? > >> > >> My expectation, if I requesting a test run for example 10 times, the > >> sequence should be as follows: > >> > >> fixture_setup, then test, then fixture_cleanup repeats 10 times > >> > >> But in practice, I've got following sequence > >> > >> 1. fixture_setup runs 10 times > >> 2. test runs 10 times > >> 3. fixture_cleanup runs 10 times > >> > >> Is it expected behavior? > >> I'm using ctest version 3.14.2 > > > > > > The documentation for the --repeat-until-fail option says "Require each > test to run <n> times without failing in order to pass". A fixture setup or > cleanup test is still a test, so they will also be run <n> times. The logic > that implements repeating tests sets up a counter on each test and it runs > that test <n> times before marking that test as complete. This is why you > see fixture_setup run 10 times, then test runs 10 times and lastly > fixture_cleanup runs 10 times. > > > > I understand that fixture is still a test. > To achieve repeatable behavior -R "test_continues_*" is enough. > > But if it calls FIXTURE why not apply the specific behavior as > expected from fixture (init/teardown)? > It's mostly related to the way it is implemented internally. Because a fixture setup/cleanup is still just another test, it has all the features of a regular test, including honouring the option that tells ctest to run it multiple times. For some projects, it may be desirable for the fixture setup/cleanup to run only once but re-run the fixture-requiring test(s) to run multiple times. For other projects, it might be desirable for the fixture setup/cleanup tests to also run multiple times (a setup test might simply be polling a service to confirm it is up, for example). Since ctest cannot know which of the two scenarios the project wants, it simply treats all tests the same, whether they are fixture setup/cleanup tests or not. Perhaps one way of dealing with this might be to add a new boolean test property which, when set, has the meaning of "only run once" when the --repeat-until-fail option is used. -- Craig Scott Melbourne, Australia https://crascit.com Get the hand-book for every CMake user: Professional CMake: A Practical Guide <https://crascit.com/professional-cmake/>
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
