I haven't seen similar issues but I have a hypothesis: make is
single-threaded and therefore it consumes output from jobs in the same
thread it uses to reap/spawn new jobs. If make is spending a large enough
amount of time consuming output then this will impacts the rate at which it
can spawn/reap new jobs.
I ran some tests to see when make attempts to consume output from child
processes and here's what I learned:
(a) For jobs executing within a single instance of make (not recursive) it
doesn't appear to begin consuming text until a recipe completes (I only
tested this with *--output-sync=target*; I expect similar to be true of
other sync modes excluding *none*).
(b) If you use *--output-sync=target* and a recipe contains the string
*${MAKE}* at all -- even if it doesn't actually start a recursive make
instance -- the way it consumes output is different (more on that below)
(c) make will not stop reading job output until there's nothing left to
consume. During that time it cannot reap/spawn new jobs. If your jobs
produce a large enough amount of text this could create enough small delays
that aggregate into the noticeable delays you're seeing.
As an example to demonstrate (b):
.RECIPEPREFIX := <
SHELL := bash
OUTER := $(shell echo -n {0..15})
INNER := $(shell echo -n {16..32})
all: outer
outer: ${OUTER}
inner: ${INNER}
${OUTER}:
<@${MAKE} -f /tmp/makefile inner
${INNER}:
<@perl -e 'my $$tmp = "${MAKE}"; map { printf "x\n"; sleep 1 } (0..100)'
I executed with (make 4.1):
make -j64 --output-sync=target
... and it begins printing output almost immediately in spite of the fact
none of the recipes could have finished. If you take *${MAKE}* out of the
recipe then it behaves as expected.
Now consider what I said w/regard to (c): it will only move on to
spawning/reaping other jobs (or consuming other inputs) if there's nothing
to read for the job it's currently consuming text from.
When I use *--output-sync=recurse* I don't see the same behavior, but if
your build recurses more than 1 level then perhaps each sub-make attempts
to consume output from its recursive make calls, possibly eating away at
time it could spend spawning/reaping jobs.
If I'm right this could be completely avoided for those of us who would
rather have log files instead of having everything dumped to the terminal.
If the parent make still needs to consume the output text, though, then the
limiting factor is: make is single-threaded and consumes input from
recursive jobs in the same thread it spawns/reaps new jobs.
-brian
_______________________________________________
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
