On Tue Feb 6 15:12:45 2018, hans.ottevan...@gmail.com (Hans Ottevanger) wrote:
> Hi, > > We are attempting to use GNU Make 4.2.1 to build a large software tree > (~290000 targets). > > The build hosts we are testing on are (virtualized) quad core X86-64 > machines running RHEL 6. Our code tree is currently on "locally > perceived" storage. We compiled GNU Make 4.2.1 from sources. > > Our main concern at this moment is the time needed for "nil > incrementals", i.e. the time needed to establish that nothing has > changed, so nothing needs to be done. If we put all dependencies in > the Makefile itself or in one single included makefile, we manage to > do a nil incremental in about 12 seconds. > > But we plan to use auto-generated dependencies and thus include a .d > file in our main Makefile for each target. Currently auto dependency > generation is possible for about 90000 of our targets. Nil incremental > builds cost increases to about 140 seconds in that case, which is > quite unacceptable for our usage. We found that this overhead > increases approximately quadratically with the number of included > dependency files. > > It appears that this behavior is caused by the need to maintain the > variable MAKEFILE_LIST. The names of all included makefiles are > appended one by one to the ever growing string representing this > variable.This also explains why memcpy() and strlen() champion the > profiled runs we did and also the quadratic behavior we observed. When > we change the source (read.c) so that the MAKEFILE_LIST is not > maintained at all, the overhead is gone and we are back to about 12 > seconds again. > > Are there any ideas (or opinions) about how to get rid of this type of > undesirable overhead? One of our own ideas is to provide a special > variant of -include (called e.g. dinclude?), intended to include just > .d files, without maintaining MAKEFILE_LIST. On the other hand it > might be feasible to improve the efficiency of appending to a > variable. Any suggestions? The latter should be possible by accumulating the elements in a hash table or linked list and composing the resulting string afterwards, allocating it once and strcpying the elements into position. Meanwhile, I think having a $MAKEFILE_LIST value of over a million characters will cause problems when you try to use it. So it does seem to be a good idea to add some kind of option for curbing or preventing its creation. -- Reinier Post TU Eindhoven _______________________________________________ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
