Hi Eric,
I noticed something that may relates to the problem. Like I said in my previous
mail, add_test is a call to cmake:
ADD_TEST(NAME ${TARGET_NAME}
…
COMMAND ${CMAKE_COMMAND}
-DPROGRAM1
-DPROGRAM2
…
Is there a chance that PROGRAM1 and PROGRAM2 are executed asynchronically? Here
is what I noticed. If a test is executed from its own binary directory, ‘ctest
-VV’, it always pass. If the same test is executed from the main binary
directory, ‘ctest -VV -R zamod’, it fails because PROGRAM2 is executed before
PROGRAM1 is finished. Attached are the screen outputs for both runs.
Thanks,
Ted
================================================================
cd CTEST/zamod
s1002807:zamod tesari$ ctest -VV
UpdateCTestConfiguration from
:/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl
UpdateCTestConfiguration from
:/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl
Test project /Users/tesari/build/gnu.dbg/CTEST/zamod
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: zamod_serial
1: Test command: /Applications/CMake.app/Contents/bin/cmake "-D"
"PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D"
"OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D"
"PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&
/Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D"
"PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
"-P" "run_and_assert.cmake"
1: Test timeout computed to be: 9.99988e+06
1: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&
/Users/tesari/build/gnu.dbg/bin/kodiak zamod"
1: -- Executing
"python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
1: /Users/tesari/kodiak/trunk/benchmark/Scripts
1: Number of arguments: 4 arguments.
1: Argument List:
['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py',
'/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk',
'/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']
1: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with 3
arguments:
1: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk
1: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk
1: 3 0
1: zamod Q0.4 PASS
1: zamod Q0.5 PASS
1: Kodiak ctest: SUCCESS
1/1 Test #1: zamod_serial ..................... Passed 5.73 sec
================================================================
s1002807:gnu.dbg tesari$ ctest -VV -R zamod
UpdateCTestConfiguration from
:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
UpdateCTestConfiguration from
:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
Test project /Users/tesari/build/gnu.dbg
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 113
Start 113: zamod_serial
113: Test command: /Applications/CMake.app/Contents/bin/cmake "-D"
"PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D"
"OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D"
"PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&
/Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D"
"PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
"-P" "run_and_assert.cmake"
113: Test timeout computed to be: 1500
113: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&
/Users/tesari/build/gnu.dbg/bin/kodiak zamod"
113: -- Executing
"python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
113: /Users/tesari/kodiak/trunk/benchmark/Scripts
113: Number of arguments: 4 arguments.
113: Argument List:
['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py',
'/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk',
'/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']
113: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with 3
arguments:
113: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk
113: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk
113: 3 0
113: /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk does not exist!
From: Eric Noulard [mailto:[email protected]]
Sent: Wednesday, May 17, 2017 10:54 PM
To: T.Sariyski <[email protected] <mailto:[email protected]> >
Cc: CMake ML <[email protected] <mailto:[email protected]> >
Subject: Re: [CMake] CTest question
We lack information on how your test are failing. If they all timeout then may
be you should setup a proper timeout for the test. The actual execution speed
of a test may depend on the load of your machine or the network (depending on
what the test are actually doing).
This may be even more sensible if your test are run in parallel. Do you run
ctest with --parallel option?
Do your tests depends on each other? If so did you setup test dependencies?
Le 17 mai 2017 10:59 PM, "T.Sariyski" < <mailto:[email protected]>
[email protected]> a écrit :
Hi,
I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’
and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the
number of these false failures is about one third. I guess that the problem is
in the way I set my tests, but I have no clue where to look, or how to ‘debug’
the ctest run.
Any advice is highly appreciated.
Thanks,
Ted
--
Powered by www.kitware.com <http://www.kitware.com>
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
