(0035792)
Peter Kuemmel (developer)
2014-04-27 12:30

Any explanation for this? Do you see this also when the fs-cache is hot?

(0035793)
Andrey Morozov (reporter)
2014-04-27 12:45

Ok. When we have a lot of little files and sever a big files this problem is actual.

For example: we have 4 file with build time 4sec, 3sec, 10sec, 2sec and two core(for build).So.

not sorted target:
1thread: 4s + 10s = 14s
2thread: 3s + 2s = 5s
total time: 14s
(one thread is sleep around 9 sec - this is a BIG problem)

But if all target will be sorted (10sec, 4sec, 3sec, 2sec)

sorted target:
1thread: 10s = 10s
2thread: 4s + 3s+ 2s = 9s
total time is 10s.

I understand that the build time and file size are not related, but it's better than nothing.

(0035794)
Peter Kuemmel (developer)
2014-04-27 14:11

Ninja supports "pools":
http://martine.github.io/ninja/manual.html#ref_pool [^]

Which are available in next cmake version:

http://www.cmake.org/cmake/help/git-next/prop_gbl/JOB_POOLS.html#prop_gbl:JOB_POOLS [^]

You could add all expensive files to one target (e.g. a static lib) and limit the parallel compiles for this target, this guaranties at least that not two expensive files are compiled in parallel. Not sure if it solves completely your problem.

(0035795)
Andrey Morozov (reporter)
2014-04-27 16:39

I think it's not solve my problem.

The solving is just sorted target by file size(first is biggest) before write to build.ninja

(0035797)
Ben Boeckel (developer)
2014-04-28 00:04

File size is not a good indicator of compile speed. C compiles faster than C++ and non-templated C++ code compiles faster than templated C++ code. Not to mention that headers may hide thousands of lines of code.

Ninja keeps stats on compilation (at least the command line and time). Adding the time it took might to it then have ninja do some naive bin-packing to try and pack the compile jobs into the available parallelism.

(0035798)
Peter Kuemmel (developer)
2014-04-28 02:26

Yes, for instance have a look at this 16kB file

http://www.cs.rpi.edu/~musser/design/blitz/fft.html [^]

By setting an arbitrary large size of the FFT you could get any compile time (at least until the compiler dies or you run out of memory).

(0035799)
Andrey Morozov (reporter)
2014-04-28 03:25

then we need a tool to determine the order (priority) of the assembly.

(0035800)
Ben Boeckel (developer)
2014-04-28 12:17

I don't think this is CMake's job. Also, if you want to rescan the files, you'll need to rerun CMake to get the new build estimates; effectively making the configure/generate step depend on every source file. Also, how would you order files between different libraries? Link targets should also be ordered.

Sorry, I don't see why this isn't better done in Ninja itself since it has *all* the information and can get actual timings rather than just guessing. It also helps non-CMake projects using Ninja.

(0035818)
Ben Boeckel (developer)
2014-05-05 16:06

Bug filed for ninja: https://github.com/martine/ninja/issues/759 [^]

(0042542)
Kitware Robot (administrator)
2016-06-10 14:29

Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.

Notes
(0035792) Peter Kuemmel (developer) 2014-04-27 12:30	Any explanation for this? Do you see this also when the fs-cache is hot?

(0035793) Andrey Morozov (reporter) 2014-04-27 12:45	Ok. When we have a lot of little files and sever a big files this problem is actual. For example: we have 4 file with build time 4sec, 3sec, 10sec, 2sec and two core(for build).So. not sorted target: 1thread: 4s + 10s = 14s 2thread: 3s + 2s = 5s total time: 14s (one thread is sleep around 9 sec - this is a BIG problem) But if all target will be sorted (10sec, 4sec, 3sec, 2sec) sorted target: 1thread: 10s = 10s 2thread: 4s + 3s+ 2s = 9s total time is 10s. I understand that the build time and file size are not related, but it's better than nothing.

(0035794) Peter Kuemmel (developer) 2014-04-27 14:11	Ninja supports "pools": http://martine.github.io/ninja/manual.html#ref_pool [^] Which are available in next cmake version: http://www.cmake.org/cmake/help/git-next/prop_gbl/JOB_POOLS.html#prop_gbl:JOB_POOLS [^] You could add all expensive files to one target (e.g. a static lib) and limit the parallel compiles for this target, this guaranties at least that not two expensive files are compiled in parallel. Not sure if it solves completely your problem.

(0035795) Andrey Morozov (reporter) 2014-04-27 16:39	I think it's not solve my problem. The solving is just sorted target by file size(first is biggest) before write to build.ninja

(0035797) Ben Boeckel (developer) 2014-04-28 00:04	File size is not a good indicator of compile speed. C compiles faster than C++ and non-templated C++ code compiles faster than templated C++ code. Not to mention that headers may hide thousands of lines of code. Ninja keeps stats on compilation (at least the command line and time). Adding the time it took might to it then have ninja do some naive bin-packing to try and pack the compile jobs into the available parallelism.

(0035798) Peter Kuemmel (developer) 2014-04-28 02:26	Yes, for instance have a look at this 16kB file http://www.cs.rpi.edu/~musser/design/blitz/fft.html [^] By setting an arbitrary large size of the FFT you could get any compile time (at least until the compiler dies or you run out of memory).

(0035799) Andrey Morozov (reporter) 2014-04-28 03:25	then we need a tool to determine the order (priority) of the assembly.

(0035800) Ben Boeckel (developer) 2014-04-28 12:17	I don't think this is CMake's job. Also, if you want to rescan the files, you'll need to rerun CMake to get the new build estimates; effectively making the configure/generate step depend on every source file. Also, how would you order files between different libraries? Link targets should also be ordered. Sorry, I don't see why this isn't better done in Ninja itself since it has all the information and can get actual timings rather than just guessing. It also helps non-CMake projects using Ninja.

(0035818) Ben Boeckel (developer) 2014-05-05 16:06	Bug filed for ninja: https://github.com/martine/ninja/issues/759 [^]

(0042542) Kitware Robot (administrator) 2016-06-10 14:29	Resolving issue as `moved`. This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.

Issue History
Date Modified	Username	Field	Change
2014-04-27 08:28	Andrey Morozov	New Issue
2014-04-27 12:30	Peter Kuemmel	Note Added: 0035792
2014-04-27 12:45	Andrey Morozov	Note Added: 0035793
2014-04-27 14:11	Peter Kuemmel	Note Added: 0035794
2014-04-27 16:39	Andrey Morozov	Note Added: 0035795
2014-04-28 00:04	Ben Boeckel	Note Added: 0035797
2014-04-28 02:26	Peter Kuemmel	Note Added: 0035798
2014-04-28 03:25	Andrey Morozov	Note Added: 0035799
2014-04-28 12:17	Ben Boeckel	Note Added: 0035800
2014-05-05 16:06	Ben Boeckel	Note Added: 0035818
2016-06-10 14:29	Kitware Robot	Note Added: 0042542
2016-06-10 14:29	Kitware Robot	Status	new => resolved
2016-06-10 14:29	Kitware Robot	Resolution	open => moved
2016-06-10 14:29	Kitware Robot	Assigned To	=> Kitware Robot
2016-06-10 14:31	Kitware Robot	Status	resolved => closed

Relationships