You could approach it in a different way: Rather than having the end user running ctest control what tests are run with -I or -R, you could have them provide a file that lists the tests to run. Then you could construct the list from that file, run only the configure/build steps that are necessary, and then use ctest_test along with its INCLUDE argument to limit the tests to the set specified.
Just an idea. HTH, David On Mon, Oct 17, 2011 at 12:12 PM, Tim Gallagher <[email protected]> wrote: > Yes, we're writing the CTestTestfile.cmake with all of the available tests > using add_test(). So it sounds like there is no way to only configure/build > selected tests when using CTest outside of CMake, short of hand-writing the > CTestTestfile with only the desired tests. > > I was going to suggest having a BUILD_COMMAND and CONFIGURE_COMMAND option to > the add_test() call, but that sounds like that also won't fix the problem > because the add_test is after the configure/build step. Obviously those would > not be useful for projects using CMake, but it would be good for ones not > using CMake. > > The way the entire process is set up -- we have our source code built with > CMake. All of our test cases were created prior to CMake'ing things and are > fairly involved to set up, so we have that setup.py script. That script > creates the bin, input, and output directories, copies over the needed input > files, etc.. Then it modifies the input files for the specific test case and > writes out a restart file with the initial conditions. Then, in the bin > directory, it creates a build directory where it configures CMake for the > main code, and another build directory where it configures CMake for the > utilities. Then it builds them all and runs the test cases on our cluster. > The tests can be run external to ctest with: > > ./setup.py -t <test category> <test name> <python dictionary of options> > > and we were trying to wrap all of that process up with CTest and use CDash to > track the cases. But as users write more test cases, they asked if it was > possible to make sure their case worked with CTest without having to > build/run all of the other cases. > > It is like pulling teeth to just get them to add the add_test() line in the > CTestTestfile even though it's just copy-pasting what's already there, so I > was hoping there would be a way to do this without making them write custom > files because I don't think that would actually happen. But, if that's not > possible, then we'll work with what we have. > > Thanks, > > Tim > > > ----- Original Message ----- > From: "David Cole" <[email protected]> > To: [email protected] > Cc: [email protected] > Sent: Monday, October 17, 2011 11:54:57 AM > Subject: Re: [CMake] CTest outside of CMake > > On Mon, Oct 17, 2011 at 11:04 AM, Tim Gallagher > <[email protected]> wrote: >> Hi, >> >> Sorry I wasn't clearer. That section was from a script that we call with >> ctest. The entire section that runs things looks like >> >> # Load our custom settings >> ctest_read_custom_files("${CTEST_REPO_DIRECTORY}") >> ctest_start("Nightly") >> ctest_update() >> foreach(TESTCASE ${LESLIE_AVAILABLE_TESTCASES}) >> set(CTEST_CONFIGURE_COMMAND "./setup.py -t ${TESTCASE} >> ${LESLIE_CONFIGURE_DICT}") >> ctest_configure(BUILD "${CTEST_REPO_DIRECTORY}" APPEND) >> set(CTEST_BUILD_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_BUILD_DICT}") >> ctest_build(BUILD "${CTEST_REPO_DIRECTORY}" APPEND) >> endforeach() >> ctest_test(BUILD "${CTEST_REPO_DIRECTORY}") >> ctest_submit() >> >> And we call ctest with >> >> ctest -S CTestScript.cmake >> >> My question is, if I only want to run a few tests, ie only run tests 2, 3, 4: >> >> ctest -S CTestScript.cmake -I 2,4 >> >> This only run the ctest_test() on tests 2, 3, and 4, but I don't know how to >> determine inside the script which tests will be run, so it configures/builds >> all of the available tests since I have that loop. >> >> So, is there a variable that gets set or some way to determine which tests >> are called from the command line inside the script in the event something >> like -I or -R flags are used? Is there a variable that contains the list of >> tests that CTest will run? >> > > No, there is no such variable. The list of tests is not knowable until > after the configure step in a CMake-controlled project. The list of > tests is only read and known internally during the execution of the > ctest_test command. How does ctest even know the list of tests in your > case, when you are running a "setup.py" script as the configure > step...? Are you writing the CTestTestfile.txt files yourself? > > >> Hope that's clearer. Thanks, >> >> Tim >> >> ----- Original Message ----- >> From: "Eric Noulard" <[email protected]> >> To: [email protected] >> Cc: [email protected] >> Sent: Monday, October 17, 2011 3:06:48 AM >> Subject: Re: [CMake] CTest outside of CMake >> >> 2011/10/17 Tim Gallagher <[email protected]>: >>> Hi, >>> >>> We have our project set up to run a series of test cases, but we are >>> treating it as if we are not using CMake. I found a tutorial online on how >>> to set it up, and we have it running just fine when we want to run all of >>> the tests. But, I haven't figured out how to do the configure/build steps >>> for selected tests only. >>> >>> For example, each test has it's own configure command and build command. If >>> I do >>> >>> ctest -I 2,2 >>> >>> then it configures and builds all the tests, but will only run the second >>> test. >>> >>> The driver script contains a section like: >>> >>> foreach(TESTCASE ${LESLIE_AVAILABLE_TESTCASES}) >>> set(CTEST_CONFIGURE_COMMAND "./setup.py -t ${TESTCASE} >>> ${LESLIE_CONFIGURE_DICT}") >>> ctest_configure(BUILD "${CTEST_REPO_DIRECTORY}" APPEND) >>> set(CTEST_BUILD_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_BUILD_DICT}") >>> ctest_build(BUILD "${CTEST_REPO_DIRECTORY}" APPEND) >>> endforeach() >>> >>> and the LESLIE_AVAILABLE_TESTCASES is a list of test cases names that match >>> the add_test() test names. Is it possible when running CTest with -I or -R >>> (or even without -I or -R) to figure out which tests it selects so we can >>> build LESLIE_AVAILABLE_TESTCASES from that? >> >> I'm not sure to understand your whole testing process flow, >> Is the script above part of a ctest -S script? >> >> Filtering some tests may be done with test NAMES or LABELS >> see EXCLUDE / INCLUDE or >> EXCLUDE_LABEL / INCLUDE_LABEL >> arguments of ctest_test (ctest --help-command ctest_test) >> >> However you does not seem to use ctest_test ? >> How do you run your ctest command (which arguments/options are you using)? >> >> Concerning labels >> see http://www.kitware.com/blog/home/post/11 and reference therein. >> >> >> >> -- >> Erk >> Membre de l'April - « promouvoir et défendre le logiciel libre » - >> http://www.april.org >> -- >> >> Powered by www.kitware.com >> >> Visit other Kitware open-source projects at >> http://www.kitware.com/opensource/opensource.html >> >> Please keep messages on-topic and check the CMake FAQ at: >> http://www.cmake.org/Wiki/CMake_FAQ >> >> Follow this link to subscribe/unsubscribe: >> http://www.cmake.org/mailman/listinfo/cmake > -- > > Powered by www.kitware.com > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Follow this link to subscribe/unsubscribe: > http://www.cmake.org/mailman/listinfo/cmake -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
