On Mon, Dec 8, 2008 at 8:14 PM, Bill Hoffman <span dir="ltr"><<a href="mailto:bill.hoffman@kitware.com">bill.hoffman@kitware.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Furthermore, the possibility remains that CxxTest may be able to be customized to simplify the output when running within another testing system.<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Does the Testing/Temporary/LastTest.log file help at all? So, if you run tests and there are failures, you can look at that file to see the detail. Maybe if ctest did something like this:<br>
The following tests FAILED:<br>
2 - unittest_addition_fail (Failed)<br>
(See Testing/Temporary/LastTest.log for details.)<br>
</blockquote>
<br>
LastTest.log is a nice thing to have but if the user has 800 unit tests I think he'd prefer not to look through the entire logfile.<br>
<br>
</blockquote></div>
If you had 800 tests and 10 or so lines for each failed tests, I would think having them scroll by on the screen would be bad. I would rather see that there are some failures, and if I want to find them I can look in the file for them.</blockquote>
<div><br>Yes, it would definitely be a bad default for this reason (especially if they all fail) =) <br></div><div> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d"><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
How about giving the user the command to run to get information for the tests that failed?<br>
<br>
(Run "ctest --output-on-failure -N 2,5,9-11" for details about the failed tests.) <br>
Obviously -N would also need to be implemented for this to work right (-I is kinda bloated and doesn't support multiple ranges).<br>
<br>
</blockquote></div>
We already have a -N and it does something different. Maybe -I can be extended. I would hate to bloat even more with an alternate command line option that did the same thing....</blockquote><div><br>Bill,<br><br>I'm afraid that extending -I any further would yield a command that is even more complicated and confusing that it is already. Sometimes it's best to just start over, I'm afraid. Users understand what "2,5,9-11" mean in the context of running tests. Obviously -I is useful when you're doing striding but other than that it is awfully complicated and looks like it has been extended to death. Witness the support for -I <x,y,z,[a],[b],...> | Test file] where "a" and "b" are "additional tests" and "Test file" is... actually, I have no idea what "Test file" is? Three command line options, one for specifying individual tests by number or range, one for striding, and one for the "Test file" would be so much more clear than one option that tries to do everything.<br>
<br>Regarding "-N" currently being used, how about "-n" for this? CTest seems to not have any lowercase individual character arguments.<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
However, all that said, I suppose an option to output on failure would not be so bad as long as it is not the default.</blockquote><div><br>All things being equal, I actually think output on failure probably would have been a better default for most users with the current behavior of CTest being achieved through some kind of "--summarize" flag. Of course my opinion is coming as a C++ developer where if a "unit test" fails within CxxTest we're dealing with perhaps 15 lines of output. Obviously CTest does more than just unit tests. When you test CMake within CTest I can imagine some of the more complicated tests like bootstrap could be closer to 1500 lines of output (wouldn't want that one to fail) =)<br>
<br></div></div>-- <br>Philip Lowman<br>