<HTML dir=ltr><HEAD>
<META content="text/html; charset=unicode" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.19328"></HEAD>
<BODY>
<DIV><FONT color=#000000 size=2 face=Arial>I just began to make ctest replace autotools for the program I develop, but I still have some problems while running ctest.</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>Computer:<BR>Intel(R) Core(TM)2 Duo CPU<BR>E7500 @ 2.93GHz<BR>RAM 3,37 GO</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>System:<BR>Microsoft Windows XP<BR>Version 2002<BR>Service Pack 3</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>Environment:<BR>Cygwin 1.7.17(0.262/5/3)</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>cmake version:<BR>2.8.9</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>Context:<BR>Application made using cmake. Testing by ctest.<BR>Tests are run either by the command:<BR>make test<BR>or<BR>ctest</FONT></DIV>
<DIV><FONT size=2 face=Arial>Example:<BR>------------------------------------------------------------------------<BR>$ ctest<BR>Test project /cygdrive/d/F053022/Programs/middle-1.4/build<BR> Start 1: t_proto_child<BR> 1/30 Test #1: t_proto_child .................... Passed 0.09 sec<BR> Start 2: t_CsvReader<BR>...</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>18/30 Test #18: test_05 .......................... Passed 0.28 sec<BR> Start 19: test_06<BR>19/30 Test #19: test_06 .......................... Passed 0.23 sec<BR> Start 20: test_07<BR>20/30 Test #20: test_07 ..........................***Not Run 0.25 sec<BR> Start 21: test_08<BR>21/30 Test #21: test_08 ..........................***Not Run 0.27 sec<BR> Start 22: test_09<BR>22/30 Test #22: test_09 .......................... Passed 0.33 sec<BR> Start 23: test_10<BR>...</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>25/30 Test #25: test_12 .......................... Passed 0.33 sec<BR> Start 26: test_13<BR>26/30 Test #26: test_13 .......................... Passed 0.31 sec<BR> Start 27: test_14<BR>27/30 Test #27: test_14 ..........................***Not Run 0.37 sec<BR> Start 28: test_15<BR>28/30 Test #28: test_15 .......................... Passed 0.44 sec<BR> Start 29: test_16<BR>29/30 Test #29: test_16 .......................... Passed 0.53 sec<BR> Start 30: test_17<BR>30/30 Test #30: test_17 ..........................***Not Run 0.39 sec</FONT></DIV>
<DIV><FONT size=2 face=Arial>87% tests passed, 4 tests failed out of 30</FONT></DIV>
<DIV><FONT size=2 face=Arial>Total Test time (real) = 7.86 sec</FONT></DIV>
<DIV><FONT size=2 face=Arial>The following tests FAILED:<BR> 20 - test_07 (BAD_COMMAND)<BR> 21 - test_08 (BAD_COMMAND)<BR> 27 - test_14 (BAD_COMMAND)<BR> 30 - test_17 (BAD_COMMAND)<BR>Errors while running CTest</FONT></DIV>
<DIV><FONT size=2 face=Arial>------------------------------------------------------------------------</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>The tests that fail make it randomly. That is, for instance, if I run,<BR>for the test test_07 that failed:</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>ctest -R test_07</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>the test may succeed. Or if I run make test again, the set of tests that<BR>fail may be different.</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>All test_nn test are built on the same model. for instance:</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>add_test (test_07 ${PROJECT_SOURCE_DIR}/test/test_driver<BR> -N 07 -S 0 -P test_parser)</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>As the tests were run originally from automake, an environment variable<BR>is set, at end of CMakeLists.txt file by the command:</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>set_tests_properties (<BR> test_01 test_02 test_03 test_04 test_05<BR> test_06 test_07 test_08 test_09 test_10<BR> test_11 test_12 test_13 test_14 test_15<BR> test_16 test_17<BR> PROPERTIES ENVIRONMENT srcdir=${PROJECT_SOURCE_DIR}/test)</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>A random failure of the tested program is discarded, because the tested<BR>program is activated through a shell, "test_driver".</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>This shell outputs a trace upon entry and exit.</FONT></DIV>
<DIV><FONT size=2 face=Arial>The final trace on exit is produced by the following script lines (korn shell). If<BR>the trace produced contains "PASS", I don't see how the exit status could<BR>differ from 0:</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial>if [[ ${errors} -gt 0 ]] ; then<BR> MSG=FAIL<BR>else<BR> MSG=PASS<BR>fi<BR>echo "${MSG}" >> trc_${NUM}.txt<BR>exit ${errors}</FONT></DIV><FONT size=2 face=Arial>
<DIV><BR>All tests mentioned as "Not Run" are actually run, since a corresponding<BR>trace output file is found for each test.</DIV>
<DIV> </DIV>
<DIV>In all cases, the trace output is normal, indicating correct entry and<BR>a normal status on exit, just before the exit command, whith the 0<BR>parameter that would normally correspond to success.</DIV>
<DIV> </DIV>
<DIV>Thus, the problem seems to come from ctest itself.<BR>------------------------------------------------------------------------</DIV>
<DIV>On the internet, typing "cmake random BAD_COMMAND error" gets some stories,<BR>but with no interesting answer.</DIV>
<DIV> </DIV>
<DIV>Is this king of behavior already known ?<BR>Does that correspond to a bug ?<BR>Is there a known workaround ?</DIV>
<DIV><BR>andre</FONT></DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV>
<DIV><FONT size=2 face=Arial></FONT> </DIV></BODY></HTML>