Hm, yours was a use case we didn't really consider when we were making changes to cost behavior. The middle ground here would be to respect costs in the non-parallel case when they are expressed explicitly, but not to cost-order them automatically based on their previous run times.<div>
<br></div><div>-Zach<br><br><div class="gmail_quote">On Tue, Nov 30, 2010 at 12:43 PM, Tyler Roscoe <span dir="ltr"><<a href="mailto:tyler@cryptio.net">tyler@cryptio.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Fri, Nov 26, 2010 at 10:38:44AM -0500, Zach Mullen wrote:<br>
> I just realized why this isn't working -- it's actually not a regression.<br>
<br>
</div>Maybe we have different definitions of "regression". I see a feature<br>
that used to do one thing but which now does something else.<br>
<br>
Here is what the docs say about the COST property:<br>
<br>
# COST: Set this to a floating point value. Tests in a test set will be<br>
# run in descending order of cost.<br>
<br>
This property describes the cost of a test. You can explicitly set this<br>
value; tests with higher COST values will run first.<br>
<br>
I don't see anything there about parallel or non-parallel runs. It seems<br>
to me that if I set the COST property, I should be able to control the<br>
order in which tests run, period. So at the very least, the docs should<br>
be updated if you intend to change the behavior.<br>
<div class="im"><br>
> In this release we decided that the costs should only be taken into account<br>
> in a parallel case (ctest -j N). Many users have implicit dependencies<br>
> based on the order of their add_test calls, so we didn't want to break<br>
> backward compatibility for those not using parallel ctest.<br>
<br>
</div>It looks like ctest -j2 is respecting COST. Currently I have several<br>
tests that cannot run at the same time as others (they touch the same<br>
resources and/or running two of them at once would crush the machine).<br>
If I could get the old COST behavior by running ctest -j1, that might be<br>
an acceptable workaround, but it does not appear to work today.<br>
<div class="im"><br>
> The non-parallel way to specify a test to run last is simply to make it the<br>
> last add_test call.<br>
<br>
</div>My CMake projects are modular (I imagine this is true for many CMake<br>
users). Each module is responsible for adding its own unit tests and<br>
code quality checks. As I said in my initial email, the code quality<br>
checks must run after the unit tests so that accurate code coverage<br>
values can be calculated. I can try to insure that my add_unittest()<br>
functions all run before my add_code_quality() functions, but that seems<br>
brittle and error-prone. It was much nicer when I could just tell<br>
add_code_quality() to add all its tests with COST -1000 to guarantee<br>
they run after everything else.<br>
<br>
I can imagine ways to work around this problem, but they all seem rather<br>
clunky, especially when COST used to solve the problem so simply and<br>
elegantly.<br>
<br>
I hope we can reach a useful middle ground about the future of the COST<br>
property. In its current state, it is of no use to me.<br>
<br>
Thanks,<br>
<font color="#888888">tyler<br>
</font><div><div></div><div class="h5"><br>
> On Fri, Nov 26, 2010 at 10:20 AM, Zach Mullen <<a href="mailto:zach.mullen@kitware.com">zach.mullen@kitware.com</a>>wrote:<br>
> > On Tue, Nov 23, 2010 at 6:02 PM, David Cole <<a href="mailto:david.cole@kitware.com">david.cole@kitware.com</a>>wrote:<br>
> ><br>
> >> It might be due to this commit:<br>
> >><br>
> >> <a href="http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=142edf8ad4baccd991a6a8a3e5283d0b575acca2" target="_blank">http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=142edf8ad4baccd991a6a8a3e5283d0b575acca2</a><br>
> >> (first released in 2.8.3)<br>
> >><br>
> >> Or this one:<br>
> >><br>
> >> <a href="http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=b4d27dc041c9164d6f3ad39e192f4b7d116ca3b3" target="_blank">http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=b4d27dc041c9164d6f3ad39e192f4b7d116ca3b3</a><br>
> >> (first released in 2.8.2)<br>
> >><br>
> >> Either way, seems like a bug to me. If you explicitly specify a COST<br>
> >> property value, especially a negative one to induce "last run" status,<br>
> >> then it should be honored over either historical average measurement<br>
> >> or "failed last time, so run it first this time" behavior.</div></div></blockquote></div><br>
</div>