Hello,
I have looked through the mail archives and the manual (to some degree of
thoroughness) and have not found this idea yet. If, it exists, I apologize for
the spam in advance.
I use make with autotools in multiple directories and have observed that
parallel builds are limited to each directory, as autotools invoked make
separately for each directory. This means that if one of the targets in a
directory takes a long time to produce, many of the cores sit idle and the next
directory does not get processed. Of course, this is needed sometimes, since
the next directory may depend on the one being make'd. However, 1) most of the
time this is not the case and 2) even if it were, these dependencies may
actually be precisely known (autotools tracks these dependencies well). The
problem, of course can be solved, by having a single Makefile for the many
directories with a single invocation of make, but that is not always practical
or easy.
So here is the idea. What if a subsequent invocation of make (in a subdir)-
instead of building the target it is given - would just parse the makefile,
create a full ruleset internally and inject this ruleset into the parent make's
ruleset. Then the subsequent invocation can simply return and let the parent
instance of make actually complete the build. This would enable correct
parallelization with full respect of the dependencies. This can, of course only
be done if the subsequent invocation of make is the last thing its action is
doing. (Or one can somehow split the action to run the remaining part after the
subsequent make's target is complete.)
Of course, this is not possible if the subdirectory creates artifacts on which
subsequent directories depend, but without expressing that dependency. As far I
am concerned, this is bad practice, but in this case we should simply disable
this behaviour.
One way to implement this idea would be the parent make to open a pipe and pass
its name to subsequent invocations of make via a command-line switch.
Subsequent invocation would then dump the parsed rulesets into that pipe and
exit. The ruleset, of course can contain environment variables and other
context that may be different from that of the parent - this has to be taken
care of. I do not know anything about how make represents the complete ruleset
internally, so I have no idea, how complex this feature would be.
For example, let us assume we have 3 Makefiles, one main and one in a
subdirectory and Makefile and sub1/Makefile and sub2/Makefile
Makefile contains:
all: subs main.o
...link with sub1/lib1.o sub2/lib2.o
main.o: main.c sub1/lib1.h sub2/lib2.h
...compile
subs:
cd sub1
make
cd ../sub2
make
While sub{1,2}/Makefile would contain
all: lib{1,2}.o
sub{1,2}.o: sub{1,2}.c sub{1,2}.h
...compile
If we invoke make on the top makefile, it will compile the two lib{1,2}.o
object files serially. With the proposed feature, the two invocations of make
in the "subs" target would return immediately, by simply injecting their rules
into the top make's ruleset. The top make would correctly figure out that in
order to do "subs" it has to build the "all" of both subdirs and would begin
linking only after that.
Any comments? Is this a good idea? Can it be done?
Thanks,
Zoltan
