View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008466CMakeCTestpublic2009-02-04 07:422014-06-02 08:38
ReporterMarcel Loose 
Assigned ToRolf Eike Beer 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusclosedResolutionfixed 
PlatformOSOS Version
Product VersionCMake-2-6 
Target VersionCMake 3.0Fixed in VersionCMake 3.0 
Summary0008466: Provide finer control than pass/fail for a test program
DescriptionIn some situations you may want to skip the results of one or more tests. In order to be able to postpone that decision until running 'make test' or 'ctest', it should be possible to filter, e.g., on return value.
For example, the GNU Autotools interpret a return value 77 as 'test skipped'.
So, besides the WILL_FAIL property, I think something like SKIPPED_WHEN would be very useful.

Example: Suppose you have a test program that connects to a database. What should happen when you run the test program while the database in not available? In my opinion, there are (at least) two options: FAIL or SKIP. That decision is up to the designer of the test program. The test program might return an exit status 77 when failing to connect to the database, thus indicating that the test (results) should be skipped.
TagsNo tags attached.
Attached Filespatch file icon 0001-allow-to-mark-a-test-as-Not-Run-with-a-specific-retu.patch [^] (3,781 bytes) 2012-09-21 12:05 [Show Content]

 Relationships
related to 0013825closedKitware Robot ctest shouldn't report skipped tests as failed 

  Notes
(0017515)
Bill Hoffman (manager)
2009-09-14 21:24

How would you specify the WHEN part? Can you give an example?
(0017524)
Marcel Loose (developer)
2009-09-15 03:57

Well, I don't have any clear thoughts on this. Maybe it should even be SKIPPED_IF, instead of SKIPPED_WHEN.

The main issue I wanted to raise is that the outcome of a test program is not always black or white, it could be gray sometimes (e.g., unable to run, due to some external cause: test skipped)

Anyway, would something like this work?

set_test_properties(outputTest PROPERTIES
                    SKIPPED_IF RETURN_VALUE EQUAL 77)

Here the property SKIPPED_IF takes a boolean expression as argument. Of course, you can debate about the exact syntax, but this is the general idea.

And to stay consistent with the existing properties, maybe a SKIPPED_IF_REGULAR_EXPRESSION is needed as well.
(0022019)
David Cole (manager)
2010-08-31 11:28

We will not be addressing this for the upcoming CMake 2.8.3 release
(0023139)
David Cole (manager)
2010-11-10 12:00

ctest already provides for determining if a test passes or fails (black or white, 0 or 1) by the following mechanisms:
- return value of the called program: 0 is pass, non-0 is fail
- invert the sense of return value: 0 is fail, non-0 is pass
- match output against regular expression: match for passing, or match for failing

- it also already provides for "not run" tests in the event that the executable to be run cannot be found at test time (for example)

In order for this to be a useful feature request, we would need to define the whole scope of what else test results could mean.

In addition to pass, fail, and "not run", what other results do you want?
How would you implement that while maintaining backwards compatibility?
How would the results be indicated on CDash? (as is, or would it require CDash code changes, too?)

I think this goes well beyond the scope of what's reasonable to achieve by the 2.8.4 release unless somebody volunteers the time to dedicate to it, and considers all of these issues...

Closing for insufficient definition. Feel free to re-open to add more definition.
(0023197)
Marcel Loose (developer)
2010-11-12 04:34

I definitely see a use case here. As I already wrote in the original description, there are cases when a test program simply cannot be run successfully. My example being a program that connects to a database and the database being down.

The category "skipped" should IMO be a subcategory of "not run". It could simply be reported as "not run" on the Dashboard. If really necessary, an extra subcategory "skipped" could be added as a separate column, but I don't think that's really needed.

You already mention that "not run" is used when the executable program cannot be found. You could consider to stretch the definition of "not run" to every situation where "not all prerequisites to run the test are met", instead of just the case that the test program cannot be found.
(0023199)
David Cole (manager)
2010-11-12 07:54

How can we define "test prerequisites" such that:
- it's easy to define in terms of test properties in the CMakeLists.txt files
- ctest can tell if a prerequisite is met or not

We recently added the test property 'REQUIRED_FILES':
http://cmake.org/cmake/help/cmake-2-8-docs.html#prop_test:REQUIRED_FILES [^]

If one of those files does not exist, we treat is as a "not run" sort of test failure and do not even attempt to run the test.

Does that help?

Or do you still think we need something else in ctest?
(0023200)
Rolf Eike Beer (developer)
2010-11-12 08:10

I think giving the test program itself the possibility to say "I can't run because $foo is missing" would be indeed very useful.

The missing stuff could basically be anything. For Marcel it's a database (maybe on a different host). We drive tests with that to do some image processing with external pattern generators and whatnot. So missing could be anything like:

-there is no external generator connected
-the external generator can't do $foo what is needed in this test (there are some versions of them around)
-the board has the wrong firmware version

None of these things is critical in a way to us so we say "this did not work, we can't release", it's more like "some dude messed up his setup". Finding things as "not run" would be what we like to see there. Messing up the setup could also be "did not install correct version of $dependency so some test executable can not run" which is what we currently have.

A property that would define either a regex or a return value to be treated as "not run" for a test would be all we need.
(0023202)
Marcel Loose (developer)
2010-11-12 09:52

Earlier, I suggested something like:

set_test_properties(outputTest PROPERTIES
                    SKIPPED_IF RETURN_VALUE EQUAL 77)

Testing on return values is much easier than using a regex. Furthemore, since the author of the test program and the CMakeLists.txt file will decide on the specific return value to treat as "skipped", there's no such thing as a "magic return value" like the Autotools have.

However, if people would also like to be able to specify a regex, it could be something along the lines of:

set_test_properties(outputTest PROPERTIES
                    SKIPPED_REGULAR_EXPRESSION <regex>)
(0024863)
David Cole (manager)
2011-01-18 11:18

Looking at "older" "feedback" status bugs today... is this still an issue that we want to address moving forward? If so, please remove the "feedback" status and just set it to "assigned"... If not, please resolve it.

Thanks.
(0024903)
Marcel Loose (developer)
2011-01-19 03:29

Yes, I still think it would be very useful to have this feature implemented. I've assigned it to you, David, since you wrote the last note.
(0024904)
Sylwester Arabas (reporter)
2011-01-19 05:42

Hi, it would be perfect to have it in cmake!

We have a lot of tests that return 77 in GNU Data Language (gdl), and it would be great to have this supported by cmake as well (GDL can be configured both using autotools and cmake).

We use exit code 77 to indicate that a test for a given feature is skipped because GDL was compiled without that feature (e.g. Python bridge or some external file format support), or in case some external resources are needed to proceed with the test (e.g. X-server connection).

Regards,
Sylwester
(0027139)
David Cole (manager)
2011-08-01 11:15

I guess the test property being discussed here is really a map. A map of program return values to test status values.

  0 --> passed
  77 --> skipped

  anything not in the map --> failed

In addition to cmake and ctest being aware of this map, you would also somehow have to make CDash aware of the same map on a per-project basis.

Which completely changes the very top level page of CDash on a per-project basis.

You'd have to have "pass, fail, and skip" columns. Or whatever user-defined test statuses there happen to be in a project.

Or, you have to map "skip" into a "pass" or "fail" in order to keep the CDash interface to the test results coherent across projects.

I don't know, maybe a project-specific layout of the top-level CDash page makes sense, and I'm just biased against it because I don't understand what it would look like.

I do not see a short-term way to implement a fix for this issue in a reasonable manner that would still work well with CDash.

If somebody can provide me with more insight into a way to do that without having to change CDash's representation of the data in a significant way, please chime in.

Patches welcome, because this seems like a ***huuuuuuggggeee*** can of worms right underneath the surface.
(0027147)
Marcel Loose (developer)
2011-08-02 08:20

I don't think CDash has to be aware of this "exit status map". For me, it would be perfectly fine, if a skipped test shows up as "not run" on the dashboard.

Actually, I think a tri-state would suffice: pass, fail, indeterminate. Where indeterminate could mean "not run" or "skipped", or what have you. This could be accomplished by adding one test property, e.g., SKIPPED_IF; see my earlier notes.

Maybe it's my lack of under-the-hood knowledge of CMake/CTest/CDash, but I really don't see this huge can of worms.
(0029650)
David Cole (manager)
2012-06-08 12:56

More discussion on the mailing list starting with this message:

  http://www.cmake.org/pipermail/cmake/2012-June/050658.html [^]
(0033723)
simonh (reporter)
2013-08-16 03:39
edited on: 2013-08-16 03:40

I would also like to request this feature. Sometimes tests have prerequisites which aren't just the executable image and some other files.

"Database availability" has already been mentioned. We have tests which require commercially licensed software to be run in order to perform validation; sometimes the licence is already in use elsewhere. In this case, it is not possible to say whether the test passes or not.

Again, I support the tri-state solution. If the test runs to completion, it must be PASS, FAIL or NOT_RUN.

As we already have PASS_REGULAR_EXPRESSION, FAIL_REGULAR_EXPRESSION, and CDash uses "Not Run", how about NOT_RUN_REGULAR_EXPRESSION and NOT_RUN_RETURN_VALUE test properties?

(0033958)
Marcel Loose (developer)
2013-10-01 10:02

Any chance that Rolf's patch (uploaded a year ago) will be accepted?
(0034227)
alxanders (reporter)
2013-10-23 11:33

Finer control than pass/fail would be very useful to me for e.g.:
- GPU tests if the node happens to not have (the right type of) GPU(s)
- tests requiring a remote machine, which is down
- tests requiring a network connection to localhost via ssh, while an ssh key pair + authorized_keys is missing. (This also includes any MPI test on localhost.)

It is still useful to run on a test node that do not have e.g. GPUs; it may have another environment feature (e.g. OS, compiler) we'd like to test against. Right now with e.g. GPU tests, we cannot see what was really tested: either we get lots of tests failed, or lots of tests passed. In both cases the GPU tests didn't test anything.

I can imagine there is some complexity involved adding a new "Skipped" state. But just fold it under "Not Run" as that is what it is. You have already extended it before (with REQUIRED_FILES). We just need to be able to trigger "Not Run" through an exit status (e.g. autotools' 77 or specifiable) from an already started test. Of course, then all Passed or Not Run could be considered fine, while currently cmake always exits with status 8 on Not Run (or Failed).

So what about Rolf Eike Beer's patch? Isn't that already close?
(0034393)
Burlen (reporter)
2013-11-05 11:05

I'd like to see this feature implemented, I think it will improve tracking OpenGL driver bugs affecting VTK.

When a test is run on a driver with known bugs, the test is skipped internally, and shows up as passed in the dashboard. I suspect that it's done this way to avoid polluting the dashboard, and it's probably a good idea to keep it like this. However, I think it's an issue that it's difficult to tell which tests are being skipped on which systems and it would be very useful if these tests showed up in the dashboard as not run rather than passed.
(0034947)
Rolf Eike Beer (developer)
2014-01-15 11:50

Fix is in master now:

http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=3a0d164bb2fa03966cdd7a17c0ce63b4d54f05b7 [^]
(0036083)
Robert Maynard (manager)
2014-06-02 08:38

Closing resolved issues that have not been updated in more than 4 months.

 Issue History
Date Modified Username Field Change
2009-02-04 07:42 Marcel Loose New Issue
2009-02-12 13:06 Bill Hoffman Status new => assigned
2009-02-12 13:06 Bill Hoffman Assigned To => Bill Hoffman
2009-09-14 21:24 Bill Hoffman Note Added: 0017515
2009-09-15 03:57 Marcel Loose Note Added: 0017524
2010-08-31 11:28 David Cole Note Added: 0022019
2010-11-10 12:00 David Cole Note Added: 0023139
2010-11-10 12:00 David Cole Status assigned => closed
2010-11-10 12:00 David Cole Resolution open => won't fix
2010-11-10 12:00 David Cole Fixed in Version => CMake 2.8.4
2010-11-12 04:34 Marcel Loose Note Added: 0023197
2010-11-12 04:34 Marcel Loose Status closed => feedback
2010-11-12 04:34 Marcel Loose Resolution won't fix => reopened
2010-11-12 07:54 David Cole Note Added: 0023199
2010-11-12 08:10 Rolf Eike Beer Note Added: 0023200
2010-11-12 09:52 Marcel Loose Note Added: 0023202
2011-01-18 11:18 David Cole Note Added: 0024863
2011-01-19 03:29 Marcel Loose Note Added: 0024903
2011-01-19 03:29 Marcel Loose Status feedback => assigned
2011-01-19 05:42 Sylwester Arabas Note Added: 0024904
2011-08-01 11:15 David Cole Note Added: 0027139
2011-08-02 08:20 Marcel Loose Note Added: 0027147
2011-09-05 16:41 David Cole Fixed in Version CMake 2.8.4 =>
2011-10-28 12:31 David Cole Assigned To Bill Hoffman =>
2011-10-28 12:31 David Cole Status assigned => backlog
2012-06-08 12:56 David Cole Note Added: 0029650
2012-09-21 12:05 Rolf Eike Beer File Added: 0001-allow-to-mark-a-test-as-Not-Run-with-a-specific-retu.patch
2012-12-31 15:18 David Cole Relationship added related to 0013825
2013-08-16 03:39 simonh Note Added: 0033723
2013-08-16 03:40 simonh Note Edited: 0033723
2013-10-01 10:02 Marcel Loose Note Added: 0033958
2013-10-23 11:33 alxanders Note Added: 0034227
2013-11-05 11:05 Burlen Note Added: 0034393
2014-01-15 11:50 Rolf Eike Beer Note Added: 0034947
2014-01-15 11:50 Rolf Eike Beer Assigned To => Rolf Eike Beer
2014-01-15 11:50 Rolf Eike Beer Status backlog => resolved
2014-01-15 11:50 Rolf Eike Beer Resolution reopened => fixed
2014-01-15 11:50 Rolf Eike Beer Fixed in Version => CMake 3.0
2014-01-15 11:50 Rolf Eike Beer Target Version => CMake 3.0
2014-06-02 08:38 Robert Maynard Note Added: 0036083
2014-06-02 08:38 Robert Maynard Status resolved => closed


Copyright © 2000 - 2018 MantisBT Team