Re: Parallel make across multiple connected systems.

2024-11-28 Thread Eli Zaretskii
> From: Sean Godsell > Date: Wed, 27 Nov 2024 17:00:27 -0500 > > I was wondering if anyone has any plans to make the actual 'make' command > work across multiple > connected PC systems, via networking of some kind. It could be wireless > networking, ethernet, or even > networking through thund

Re: Parallel make across multiple connected systems.

2024-11-27 Thread Henrik Carlqvist
On Wed, 27 Nov 2024 17:00:27 -0500 Sean Godsell wrote: > I was wondering if anyone has any plans to make the actual 'make' command > work across multiple connected PC systems, via networking of some kind. It > could be wireless networking, ethernet, or even networking through > thunderbolt, usb 4

Parallel make across multiple connected systems.

2024-11-27 Thread Sean Godsell
Hello any and all. I was wondering if anyone has any plans to make the actual 'make' command work across multiple connected PC systems, via networking of some kind. It could be wireless networking, ethernet, or even networking through thunderbolt, usb 4, or even fiber. All that matters is that e

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-03-13 Thread Ken Brown
On 3/13/2023 9:15 AM, Paul Smith wrote: On Sun, 2023-03-12 at 11:25 -0400, Ken Brown wrote: If I'm right, the solution would seem to be to disable the use of pselect on Cygwin when the jobserver is using a fifo.  I'll try that on a local build of make and see if I can still reproduce the problem

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-03-13 Thread Paul Smith
On Sun, 2023-03-12 at 11:25 -0400, Ken Brown wrote: > > If I'm right, the solution would seem to be to disable the use of > > pselect on Cygwin when the jobserver is using a fifo.  I'll try > > that on a local build of make and see if I can still reproduce the > > problem. > > Never mind.  My spe

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-03-12 Thread Ken Brown
On 3/11/2023 12:51 PM, Ken Brown wrote: Update: The hang occurred again.  It appears to be caused by an infinite loop starting with a call to pselect[*]. [...] If I'm right, the solution would seem to be to disable the use of pselect on Cygwin when the jobserver is using a fifo.  I'll try tha

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-03-11 Thread Ken Brown
On 3/2/2023 5:39 PM, Ken Brown wrote: I'm returning to this thread because a surprising thing happened.  I decided to try to debug the fifo problem I reported at the beginning of the thread (a hang building TeX Live on Cygwin when the jobserver used a fifo).  So I installed make 4.4.1 built wit

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-03-02 Thread Ken Brown
I'm returning to this thread because a surprising thing happened. I decided to try to debug the fifo problem I reported at the beginning of the thread (a hang building TeX Live on Cygwin when the jobserver used a fifo). So I installed make 4.4.1 built with fifos enabled (by setting CPPFLAGS=-

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-24 Thread Ken Brown
On 2/24/2023 11:29 AM, Paul Smith wrote: On Thu, 2023-02-23 at 16:17 -0500, Ken Brown wrote: Sorry, I spoke too soon.  I just went back to my original use case, in which I used the TeX Live "Build" script instead of directly invoking make, and I again got the jobserver warning. Thanks for the

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-24 Thread Paul Smith
On Thu, 2023-02-23 at 16:17 -0500, Ken Brown wrote: > Sorry, I spoke too soon.  I just went back to my original use case, > in which I used the TeX Live "Build" script instead of directly > invoking make, and I again got the jobserver warning. Thanks for the info. TL;DR: this is not an error in m

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-24 Thread Ken Brown
On 2/24/2023 9:54 AM, Paul Smith wrote: On Thu, 2023-02-23 at 16:17 -0500, Ken Brown wrote: I'm attaching that script so you can see exactly how "make" is invoked in a subshell.  I'm also attaching my build log up to the point of the warning and the Makefile in the ft-build directory in which th

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-24 Thread Paul Smith
On Thu, 2023-02-23 at 16:17 -0500, Ken Brown wrote: > I'm attaching that script so you can see exactly how "make" is > invoked in a subshell.  I'm also attaching my build log up to the > point of the warning and the Makefile in the ft-build directory in > which the warning occurred.  I've previousl

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-23 Thread Ken Brown
On 2/21/2023 5:59 PM, Ken Brown wrote: On 2/21/2023 1:26 PM, Paul Smith wrote: On Tue, 2023-02-21 at 13:11 -0500, Ken Brown wrote: I think you're on the right track.  I got through 'make -j13 check' without the jobserver warning. w00t!  Thanks for the help.  A full fix should be in 4.4.1 whic

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Ken Brown
On 2/21/2023 1:26 PM, Paul Smith wrote: On Tue, 2023-02-21 at 13:11 -0500, Ken Brown wrote: I think you're on the right track.  I got through 'make -j13 check' without the jobserver warning. w00t! Thanks for the help. A full fix should be in 4.4.1 which I hope to release this week or weekend

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Paul Smith
On Tue, 2023-02-21 at 13:11 -0500, Ken Brown wrote: > I think you're on the right track.  I got through 'make -j13 check' > without the jobserver warning. w00t! Thanks for the help. A full fix should be in 4.4.1 which I hope to release this week or weekend.

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Ken Brown
On 2/21/2023 11:34 AM, Paul Smith wrote: On Tue, 2023-02-21 at 10:36 -0500, Paul Smith wrote: But, I think I might have found the bug.  If I'm right, it's a doozy! But, as you mentioned it's not widespread because it only affects pipe jobservers.  I'm glad 4.4.1 is almost ready to go.  I'll try

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Paul Smith
On Tue, 2023-02-21 at 10:36 -0500, Paul Smith wrote: > But, I think I might have found the bug.  If I'm right, it's a doozy! > But, as you mentioned it's not widespread because it only affects > pipe jobservers.  I'm glad 4.4.1 is almost ready to go.  I'll try to > write a test case for it to see i

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Paul Smith
On Tue, 2023-02-21 at 10:12 -0500, Ken Brown wrote: > > There will likely be a lot of output. > > Here it is. Oops I got the rule wrong for the set -x. But, I think I might have found the bug. If I'm right, it's a doozy! But, as you mentioned it's not widespread because it only affects pipe jo

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Ken Brown
On 2/21/2023 9:39 AM, Paul Smith wrote: On Tue, 2023-02-21 at 08:35 -0500, Ken Brown wrote: make  check-TESTS make[4]: Entering directory '/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea' make[5]: Entering directory '/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea' OK, I see th

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Paul Smith
On Tue, 2023-02-21 at 08:35 -0500, Ken Brown wrote: > make  check-TESTS > make[4]: Entering directory > '/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea' > make[5]: Entering directory > '/home/kbrown/src/texlive/test.x86_64/Work/texk/kpathsea' OK, I see the rule that generates the [5] re

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Paul Smith
On Tue, 2023-02-21 at 08:35 -0500, Ken Brown wrote: > On 2/21/2023 3:54 AM, Ken Brown wrote: > > On 2/21/2023 12:27 AM, Paul Smith wrote: > > > Just to note, I do run the regression test suite with the > > > equivalent of "jobserver-style=pipe" (basically I force the > > > configure to believe that

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-21 Thread Ken Brown
On 2/21/2023 12:27 AM, Paul Smith wrote: On Mon, 2023-02-20 at 20:21 -0500, Ken Brown wrote: Parallel make is still not working reliably.  I've just discovered that my TeX Live build logs have several occurrences of the following warning:    jobserver unavailable: using -j1.  Add &

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-20 Thread Paul Smith
On Mon, 2023-02-20 at 20:21 -0500, Ken Brown wrote: > Parallel make is still not working reliably.  I've just discovered > that my TeX Live build logs have several occurrences of the following > warning: > >    jobserver unavailable: using -j1.  Add '+' to parent

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-20 Thread Ken Brown
On 2/19/2023 9:29 AM, Paul Smith wrote: I will change the default setting of the jobserver to use "pipe" on Cygwin, at least for now. Parallel make is still not working reliably. I've just discovered that my TeX Live build logs have several occurrences of the following warning

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-19 Thread Ken Brown
On 2/19/2023 9:29 AM, Paul Smith wrote: On Sun, 2023-02-19 at 09:17 -0500, Ken Brown wrote: So I'm not sure where to go from here on Cygwin.  Should I force Cygwin builds to use the "pipe" jobserver, as I've done for GNU/Hurd? My preference would be for you to provide a configure option, which

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-19 Thread Paul Smith
On Sun, 2023-02-19 at 09:17 -0500, Ken Brown wrote: > > So I'm not sure where to go from here on Cygwin.  Should I force > > Cygwin builds to use the "pipe" jobserver, as I've done for > > GNU/Hurd? > > My preference would be for you to provide a configure option, which > defaults to "pipe" on Cy

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-19 Thread Ken Brown
On 2/19/2023 8:49 AM, Paul Smith wrote: On Wed, 2023-02-15 at 13:57 -0500, Ken Brown wrote: One thing to keep in mind here is that your tests on Cygwin were done on Cygwin 2.9.0, in which FIFOs were very poorly supported.  For example, a FIFO couldn't have multiple readers or writers.  If GNU/Hu

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-19 Thread Paul Smith
On Wed, 2023-02-15 at 13:57 -0500, Ken Brown wrote: > One thing to keep in mind here is that your tests on Cygwin were done > on Cygwin 2.9.0, in which FIFOs were very poorly supported.  For > example, a FIFO couldn't have multiple readers or writers.  If > GNU/Hurd has similar limitations, that co

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-19 Thread Paul Smith
On Sun, 2023-02-19 at 12:44 +0100, Bruno Haible wrote: >   - 4 failures in category 'features/jobserver' >   - 2 failure in category 'functions/shell' I looked at the error logging you sent, and I think these are just a result of an incomplete/incorrect setting of the default value to "pipe". Hop

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-19 Thread Bruno Haible
Paul Smith wrote: > I've made a change that causes GNU/Hurd to not use the mkfifo()-based > jobserver, and go back to using pipe(). That's nice, because compared to the previous results - 4 failures in category 'features/archives', due to "cc: not found". - 5 failures in category 'features/jo

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-18 Thread Paul Smith
On Wed, 2023-02-15 at 13:57 -0500, Ken Brown wrote: > If I simply run 'make -j13 world' without using the Build script, it > doesn't hang.  I don't know if this provides any clue as to where the > problem is. Speaking for myself, it doesn't give me any clues. I don't see why it should matter. Bu

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-18 Thread Paul Smith
On Wed, 2023-02-15 at 19:34 +0100, Bruno Haible wrote: > > I may be misremembering but I thought that you had tried forcing > > the pipe jobserver option on GNU/Hurd and it didn't help. > > This must be a misunderstanding. I never said that. I don't even know > how to do this, within the framework

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-15 Thread Ken Brown
On 2/15/2023 1:34 PM, Bruno Haible wrote: Paul Smith wrote: And possibly also on GNU/Hurd. Cf. https://lists.gnu.org/archive/html/bug-make/2023-01/msg00107.html I may be misremembering but I thought that you had tried forcing the pipe jobserver option on GNU/Hurd and it didn't help. This mus

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-15 Thread Bruno Haible
Paul Smith wrote: > > And possibly also on GNU/Hurd. Cf. > > https://lists.gnu.org/archive/html/bug-make/2023-01/msg00107.html > > I may be misremembering but I thought that you had tried forcing the > pipe jobserver option on GNU/Hurd and it didn't help. This must be a misunderstanding. I never

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-15 Thread Paul Smith
On Tue, 2023-02-14 at 01:04 +0100, Bruno Haible wrote: > And possibly also on GNU/Hurd. Cf. > https://lists.gnu.org/archive/html/bug-make/2023-01/msg00107.html I may be misremembering but I thought that you had tried forcing the pipe jobserver option on GNU/Hurd and it didn't help.

Regression on Cygwin: Problems with parallel make in 4.4

2023-02-14 Thread Ken Brown
Several packages that used to build fine on Cygwin with parallel make now fail to build. I either get strange errors or a hang. The problems started with version 4.4 and seem to stem from the fact that make now uses a FIFO by default instead of a pipe. If I use the flag '--jobserver-

Re: Regression on Cygwin: Problems with parallel make in 4.4

2023-02-13 Thread Bruno Haible
Ken Brown wrote: > I suggest > that you provide a configure option to set the jobserver style at the > time make is built, and set it to 'pipe' by default on Cygwin. And possibly also on GNU/Hurd. Cf. https://lists.gnu.org/archive/html/bug-make/2023-01/msg00107.html Bruno

[bug #57692] Parallel make doesn't work well with grouped targets

2020-05-11 Thread Robert Sachunsky
Follow-up Comment #4, bug #57692 (project make): > There is a token in .FEATURES for this capability already Oh, pardon me, I accidentally took an old version for testing. > Notice *groupd-target* here. Splendid! Would you care for a patch/PR of the manual explaining this, with an example (alo

[bug #57692] Parallel make doesn't work well with grouped targets

2020-05-11 Thread Paul D. Smith
Follow-up Comment #3, bug #57692 (project make): There is a token in .FEATURES for this capability already: $ echo '$(info $(.FEATURES))' | ./make -f- target-specific order-only second-expansion else-if shortest-stem undefine oneshell nocomment grouped-target extra-prereqs archives jobserver out

[bug #57692] Parallel make doesn't work well with grouped targets

2020-05-10 Thread Robert Sachunsky
Follow-up Comment #2, bug #57692 (project make): Still, I believe this new syntax should really be listed in *.FEATURES* That way, makefiles which want to employ this (highly valued!) new feature, can be written in a way that older versions of make can still run them. The documentation should al

[bug #57692] Parallel make doesn't work well with grouped targets

2020-01-29 Thread Paul D. Smith
int of grouped targets is to support parallel make properly: if you only use serial make then you don't really need them. However, grouped targets were introduced in GNU make 4.3. If you're using an older version, they don't exist. In 4.2.1 and before, your makefile simply define

[bug #57692] Parallel make doesn't work well with grouped targets

2020-01-29 Thread Philipp Krause
URL: <https://savannah.gnu.org/bugs/?57692> Summary: Parallel make doesn't work well with grouped targets Project: make Submitted by: spth Submitted on: Wed 29 Jan 2020 10:42:26 AM UTC Severity:

[bug #51462] Double-colon dependencies are built serially with parallel make

2017-07-13 Thread Robert Morell
Follow-up Comment #2, bug #51462 (project make): Thanks. I suspect that the fix referred to in 47995 is: http://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0 (it even starts with "[SV 47995]"). However, no, unfortunately it doesn't fix the problem I des

[bug #51462] Double-colon dependencies are built serially with parallel make

2017-07-13 Thread Martin Dorey
Follow-up Comment #1, bug #51462 (project make): A google for that hash finds: https://savannah.gnu.org/bugs/?48057 ... in which a reversion of that commit is suggested. It's marked as a duplicate of: https://savannah.gnu.org/bugs/?47995 ... which is marked as fixed in 4.2.1, which is the ver

[bug #51462] Double-colon dependencies are built serially with parallel make

2017-07-13 Thread anonymous
URL: <http://savannah.gnu.org/bugs/?51462> Summary: Double-colon dependencies are built serially with parallel make Project: make Submitted by: None Submitted on: Thu 13 Jul 2017 09:29:18 PM UTC Severity: 3 -

RE: Parallel make Order Algorithm

2015-07-14 Thread Martin Dorey
er, preparing ingredients for a cake or generating documentation. -Original Message- From: bug-make-bounces+martin.dorey=hds@gnu.org [mailto:bug-make-bounces+martin.dorey=hds@gnu.org] On Behalf Of Dilyan Palauzov Sent: Tuesday, July 14, 2015 05:02 To: bug-make@gnu.org Subject:

Parallel make Order Algorithm

2015-07-14 Thread Dilyan Palauzov
parallel make -j 4, it first compiles all object files (for the libraries and for the executables), then it links all libraries and then it links the executables. I have the feeling that the second algorithm has a bottleneck: While compiling object files is always possible, linking of

Re: Parallel make with distributed systems

2015-05-03 Thread SF Markus Elfring
> The question is, why is it the case that these challenges, tasks, work, > and constraints need to be handled within make itself, > rather than farmed out to a separate process via the SHELL capability? Does it matter to get the accounting for the processor cores right? > Offhand I can't see an

Re: Parallel make with distributed systems

2015-05-03 Thread Paul Smith
On Sun, 2015-05-03 at 09:31 +0200, SF Markus Elfring wrote: > > I have nothing against doing more than that in theory, but before I'd > > agree to add something complex I'd need to understand where the existing > > method fails, and the new method would provide significant benefits. > > I guess th

Re: Parallel make with distributed systems

2015-05-03 Thread SF Markus Elfring
> I have nothing against doing more than that in theory, but before I'd > agree to add something complex I'd need to understand where the existing > method fails, and the new method would provide significant benefits. I guess that there are some software development challenges around the specifica

Re: Parallel make

2015-05-02 Thread Paul Smith
On Thu, 2015-04-30 at 08:18 +0200, SF Markus Elfring wrote: > > GNU make has no built-in capability to use multiple machines: > > How are the chances to integrate additional job submission systems? Well, make already HAS a hook available to anyone who wants to use a different job submission syste

Re: Parallel make

2015-04-29 Thread SF Markus Elfring
> GNU make has no built-in capability to use multiple machines: How are the chances to integrate additional job submission systems? > conceptually it may be a straightforward extension but the effort needed > to communicate between multiple systems over a network, send and receive > results reli

Re: Parallel make

2015-04-29 Thread Edward Welbourne
> We ran into similar things in Linux due to NFS, autofs, and NFS via > apparmor not scaling well when 100 compilers are trying to search for > header files through a long number of sourcedirs. I suppose this can be mitigated by using #include "path/to/file.h" in source, for paths relative to a sm

Re: Parallel make

2015-04-29 Thread Edward Welbourne
> Luckily, if you are building C or C++ code someone has already done all > the necessary work for you. I recommend you investigate the distcc > package: https://code.google.com/p/distcc/ Likewise the icecream project's icecc: https://en.opensuse.org/Icecream https://github.com/icecc/icecream I'

Re: Parallel make

2015-04-29 Thread Kruppa, Jason
Last time I looked at distcc, it still ran the c++ preprocessor on the main machine (to eliminate the risk of different machines having different configurations), and then sent the preprocessed files to the various machines at its disposal. This, in my experience, still does a huge portion of t

Re: Parallel make

2015-04-29 Thread Tim Murphy
Without wanting to turn this into a commercial/advert you might want to consider trying the Electric Cloud Huddle beta since it works with multiple machines in a convenient way and deals with the problems of getting correct parallel builds. It is also free for now at least. Sorry for that :) On 29

Re: Parallel make

2015-04-29 Thread Paul Smith
On Wed, 2015-04-29 at 13:50 -0600, Ryan P. Steele wrote: > The multithreaded version of make (-j#) is wonderful, and we have made > great use of it. Because we're dealing with some very large code, > however, it would be great to be able to parallelize compilation over > multiple machines. I ca

Parallel make

2015-04-29 Thread Ryan P. Steele
Hi, folks - I encountered a situation that I can't seem to resolve through the "make" documentation... The multithreaded version of make (-j#) is wonderful, and we have made great use of it. Because we're dealing with some very large code, however, it would be great to be able to paralleliz

[bug #30823] double-colon rules not always executed in parallel make

2013-09-22 Thread Paul D. Smith
Update of bug #30823 (project make): Status:None => Duplicate Open/Closed:Open => Closed ___ Follow-up Comment #2: I was able to reproduc

Re: Infinite loop bug with parallel make

2013-02-23 Thread Paul Smith
On Sat, 2013-02-23 at 21:02 +, Ian Lynagh wrote: > On Sat, Feb 23, 2013 at 03:45:59PM -0500, Paul Smith wrote: > > > > I've never really been clear on the purpose and use of .SECONDARY; the > > comments in both the GNU make manual and in the code seem odd to me. I > > would really appreciate

Re: Infinite loop bug with parallel make

2013-02-23 Thread Ian Lynagh
On Sat, Feb 23, 2013 at 03:45:59PM -0500, Paul Smith wrote: > > I've never really been clear on the purpose and use of .SECONDARY; the > comments in both the GNU make manual and in the code seem odd to me. I > would really appreciate anyone out there who is using this (either for > specific targe

Re: Infinite loop bug with parallel make

2013-02-23 Thread Paul Smith
On Sat, 2013-02-23 at 17:28 +, Ian Lynagh wrote: > On Sat, Feb 23, 2013 at 06:57:27AM +0200, Shachar Shemesh wrote: > > > > What I'm also interested in is why .SECONDARY made everything slow. > > I've put a cut-down makefile demonstrating this here: > http://urchin.earth.li/~ian/tmp/Makef

Re: Infinite loop bug with parallel make

2013-02-23 Thread Ian Lynagh
Hi Shachar, On Sat, Feb 23, 2013 at 06:57:27AM +0200, Shachar Shemesh wrote: > > What I'm also interested in is why .SECONDARY made everything slow. I've put a cut-down makefile demonstrating this here: http://urchin.earth.li/~ian/tmp/Makefile First run $ make setup (which creates some

Re: Infinite loop bug with parallel make

2013-02-23 Thread Ian Lynagh
On Sat, Feb 23, 2013 at 01:50:38PM +, Ian Lynagh wrote: > On Fri, Feb 22, 2013 at 09:38:22PM -0500, Paul Smith wrote: > > On Sat, 2013-02-23 at 02:32 +, Ian Lynagh wrote: > > > The problem was that our compiler generates 2 output files (foo.o and > > > foo.hi) when compiling one source file

Re: Infinite loop bug with parallel make

2013-02-23 Thread Ian Lynagh
On Fri, Feb 22, 2013 at 09:38:22PM -0500, Paul Smith wrote: > On Sat, 2013-02-23 at 02:32 +, Ian Lynagh wrote: > > The problem was that our compiler generates 2 output files (foo.o and > > foo.hi) when compiling one source file, and we had thus ended up with > > a bunch of rules like > > %.

Re: Infinite loop bug with parallel make

2013-02-22 Thread Shachar Shemesh
On 02/23/2013 04:38 AM, Paul Smith wrote: > On Sat, 2013-02-23 at 02:32 +, Ian Lynagh wrote: >> The problem was that our compiler generates 2 output files (foo.o and >> foo.hi) when compiling one source file, and we had thus ended up with >> a bunch of rules like >> %.hi: %.o ; > The right

Re: Infinite loop bug with parallel make

2013-02-22 Thread Paul Smith
On Sat, 2013-02-23 at 02:32 +, Ian Lynagh wrote: > The problem was that our compiler generates 2 output files (foo.o and > foo.hi) when compiling one source file, and we had thus ended up with > a bunch of rules like > %.hi: %.o ; The right way to declare a rule that generates multiple tar

Re: Infinite loop bug with parallel make

2013-02-22 Thread Ian Lynagh
Hi Sebastian, On Sat, Feb 23, 2013 at 12:24:11AM +0100, Sebastian Pipping wrote: > On 22.02.2013 03:31, Ian Lynagh wrote: > > > Also, is there a way to tell make not to treat any file as intermediate? > > I think that it's possible that this would work around the problem. > > If that's not possi

Re: Infinite loop bug with parallel make

2013-02-22 Thread Sebastian Pipping
On 22.02.2013 03:31, Ian Lynagh wrote: > > Hi all, > > The attached Makefile causes an infinite loop with parallel make when > using make 3.81 (on amd64/Linux): > > $ make setup > touch B.hs A.hs > sleep 2 > touch B.hi B.dyn_hi > sleep 2 >

Infinite loop bug with parallel make

2013-02-21 Thread Ian Lynagh
Hi all, The attached Makefile causes an infinite loop with parallel make when using make 3.81 (on amd64/Linux): $ make setup touch B.hs A.hs sleep 2 touch B.hi B.dyn_hi sleep 2 touch B.o B.dyn_o sleep 2 touch A.dyn_hi sleep 2 touch A.o A.dyn_o $ make

Re: Using shell wrapper for descrambling parallel make output

2011-11-14 Thread Atte Peltomäki
On Mon, Nov 14, 2011 at 11:46:32AM -0500, David Boyce wrote: > On Mon, Nov 14, 2011 at 4:29 AM, Atte Peltomäki wrote: > > Nice work. Your implementation seems much more refined than mine. Only > > one thing catches my attention; your version doesn't seem to properly > > preserve the original line

Re: Using shell wrapper for descrambling parallel make output

2011-11-14 Thread David Boyce
On Mon, Nov 14, 2011 at 4:29 AM, Atte Peltomäki wrote: > Nice work. Your implementation seems much more refined than mine. Only > one thing catches my attention; your version doesn't seem to properly > preserve the original line ordering between stdout and stderr. I suggest > solving this as I did

Re: Using shell wrapper for descrambling parallel make output

2011-11-14 Thread Atte Peltomäki
On Fri, Nov 11, 2011 at 10:24:25AM -0500, David Boyce wrote: > On Fri, Nov 11, 2011 at 9:26 AM, Atte Peltomäki wrote: > > Hi, > > > > As you know, running parallel builds with '-jX' makes the shell output > > difficult to read, since output from parallel jobs are mixed. To remedy > > this, the use

Re: Using shell wrapper for descrambling parallel make output

2011-11-11 Thread David Boyce
On Fri, Nov 11, 2011 at 9:26 AM, Atte Peltomäki wrote: > Hi, > > As you know, running parallel builds with '-jX' makes the shell output > difficult to read, since output from parallel jobs are mixed. To remedy > this, the use of a buffering shell wrapper has been suggested: > > http://cmcrossroads

Using shell wrapper for descrambling parallel make output

2011-11-11 Thread Atte Peltomäki
Hi, As you know, running parallel builds with '-jX' makes the shell output difficult to read, since output from parallel jobs are mixed. To remedy this, the use of a buffering shell wrapper has been suggested: http://cmcrossroads.com/cm-basics/12838-descrambling-parallel-build-logs I liked the i

Re: Intermittent parallel make rebuild failures

2010-10-18 Thread David Highley
"David Highley wrote:" > > "Philip Guenther wrote:" > > > > Looking backwards, this rule from your top level makefile: > > > > > $(INF_DIST_SVCS_OBJS): $(DIRS) > > > > seems bogus, given that $(INF_DIST_SVCS_OBJS) is a list of objects in > > directories _below_ the list of directories in $(DIRS

Re: Intermittent parallel make rebuild failures

2010-10-10 Thread David Highley
"Philip Guenther wrote:" > > Looking backwards, this rule from your top level makefile: > > > $(INF_DIST_SVCS_OBJS): $(DIRS) > > seems bogus, given that $(INF_DIST_SVCS_OBJS) is a list of objects in > directories _below_ the list of directories in $(DIRS). The top level > make may recurse, then

Re: Intermittent parallel make rebuild failures

2010-10-09 Thread Philip Guenther
Looking backwards, this rule from your toplevel makefile: > $(INF_DIST_SVCS_OBJS): $(DIRS) seems bogus, given that $(INF_DIST_SVCS_OBJS) is a list of objects in directories _below_ the list of directories in $(DIRS). The toplevel make may recurse, then on popping back out see that the *SOURCE DI

Re: Intermittent parallel make rebuild failures

2010-10-08 Thread David Highley
"Philip Guenther wrote:" > > On Thu, Oct 7, 2010 at 9:22 AM, David Highley > wrote: > ... > > Let us just cut to the essence of the issue we are running into, how do > > target prerequisites get processed in version 3.81 of make? > > > > Example: > > $(LIBRARY): $(INF_DIST_SVCS_OBJS) > > > > $(IN

Re: Intermittent parallel make rebuild failures

2010-10-07 Thread Philip Guenther
On Thu, Oct 7, 2010 at 9:22 AM, David Highley wrote: ... > Let us just cut to the essence of the issue we are running into, how do > target prerequisites get processed in version 3.81 of make? > > Example: > $(LIBRARY): $(INF_DIST_SVCS_OBJS) > > $(INF_DIST_SVCS_OBJS): $(DIRS) > > $(DIRS): > > .PHO

Re: Intermittent parallel make rebuild failures

2010-10-07 Thread David Highley
OK, after more testing I now believe that I have leaped to the "Island of Conclusions in the Sea of Wisdom" more than once, or more accurately "I have fallen into the unconscious incompetent syndrome." Let us just cut to the essence of the issue we are running into, how do target prerequisites get

Re: Intermittent parallel make rebuild failures

2010-10-06 Thread David Highley
Paul, Since yesterday was very hectic I decided to go back and re-run all my testing for the intermittent parallel build failure. I have now concluded that Red Hat 5.4 also does not work so the last know working version was a 3.79-x version which came on Red Hat 3 Update 8. I did retest the latest

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread David Highley
"Paul Smith wrote:" > > On Tue, 2010-10-05 at 11:07 -0700, David Highley wrote: > > The latest version of Fedora 13 does not have a 3.82 verion of make so > > the only way to test would be to build from source which I can do. It > > may take me a while since were pretty busy right now. > > Buildi

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread Paul Smith
On Tue, 2010-10-05 at 11:07 -0700, David Highley wrote: > The latest version of Fedora 13 does not have a 3.82 verion of make so > the only way to test would be to build from source which I can do. It > may take me a while since were pretty busy right now. Building make from source is pretty trivi

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread David Highley
"Paul Smith wrote:" > > On Tue, 2010-10-05 at 10:23 -0700, David Highley wrote: > > Good news, it appears that Red Hat 5.4 with GNU make 3.81-3 fixes the > > issue we are running into. Now to find the right version for Cygwin. > > I'd be really interested to know whether the current latest releas

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread Paul Smith
On Tue, 2010-10-05 at 10:23 -0700, David Highley wrote: > Good news, it appears that Red Hat 5.4 with GNU make 3.81-3 fixes the > issue we are running into. Now to find the right version for Cygwin. I'd be really interested to know whether the current latest release of GNU make (3.82) has this pro

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread David Highley
"David Highley wrote:" > > "Edward Welbourne wrote:" > > > > > Every time the subject of cache comes up its because they are always > > > wrong. > > > > well, this *is* a *bug* list - every time anything comes up here, it's > > because someone's found a bug in it ! The vast amount of the time >

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread David Highley
"Edward Welbourne wrote:" > > > Every time the subject of cache comes up its because they are always > > wrong. > > well, this *is* a *bug* list - every time anything comes up here, it's > because someone's found a bug in it ! The vast amount of the time > that the cache works just fine and make

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread Edward Welbourne
> Every time the subject of cache comes up its because they are always > wrong. well, this *is* a *bug* list - every time anything comes up here, it's because someone's found a bug in it ! The vast amount of the time that the cache works just fine and makes builds faster is (quite properly) not n

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread David Highley
"Edward Welbourne wrote:" > > > it appears from running make with the debug option that the > > top level make does not see the object file created by the lower level > > make in the time when it checks the dependencies for the library. > > sounds a *lot* like an issue with make's caching of stat

Re: Intermittent parallel make rebuild failures

2010-10-05 Thread Edward Welbourne
> it appears from running make with the debug option that the > top level make does not see the object file created by the lower level > make in the time when it checks the dependencies for the library. sounds a *lot* like an issue with make's caching of stat information; if the uppper make has lo

Intermittent parallel make rebuild failures

2010-10-04 Thread David Highley
We have a build process that had no issues in the past when we were on Red Hat 3 and an older version of Cygwin. Now our core library builds fail to link intermittently on a rebuild when building on Red Hat 5.1 and Windows using Cygwin make and Windows compilers. I did some digging into it today a

Re: Prioritizing non-dependent targets in parallel make

2010-08-23 Thread Eric Melski
Eric Melski wrote: ElectricAccelerator doesn't factor runtimes into scheduling decisions, although we have talked about doing so in the past. I spent some time investigating the possibility, most recently a couple years ago. ... I ran the simulation on a variety of real-world builds. ... I

[bug #30823] double-colon rules not always executed in parallel make

2010-08-19 Thread Paul D. Smith
Update of bug #30823 (project make): Triage Status:None => Verified ___ Follow-up Comment #1: Yes, I see this too. It definitely looks like a bug. ___

[bug #30823] double-colon rules not always executed in parallel make

2010-08-19 Thread anonymous
URL: <http://savannah.gnu.org/bugs/?30823> Summary: double-colon rules not always executed in parallel make Project: make Submitted by: None Submitted on: Do 19 Aug 2010 19:24:54 UTC Severity: 3 -

Re: [RFC]serialize the output of parallel make?

2010-08-04 Thread Eric Melski
Hambridge, Philip J (ODP) wrote: I've not been following this thread too closely, but this may be relevant: http://www.cmcrossroads.com/ask-mr-make/12909-descrambling-parallel-build-logs I wrote the linked article; for those not interested in reading it, the strategy described there is the sa

RE: [RFC]serialize the output of parallel make?

2010-08-04 Thread Hambridge, Philip J (ODP)
nscript that clearly shows the errors aligned with the source. Regards, Philip. -Original Message- From: Chiheng Xu [mailto:chiheng...@gmail.com] Sent: 30 July 2010 03:00 To: bug-make@gnu.org Subject: [RFC]serialize the output of parallel make? As parallel make are becoming more and mor

Re: Fwd: [RFC]serialize the output of parallel make?

2010-08-03 Thread Chiheng Xu
On Wed, Aug 4, 2010 at 10:41 AM, Eric Melski wrote: > ElectricAccelerator is a gmake replacement that does exactly this.  I wrote > about this feature a while back: > > http://blog.electric-cloud.com/2008/12/01/untangling-parallel-build-logs/ > > You can read more about Accelerator on the blog, or

Re: Fwd: [RFC]serialize the output of parallel make?

2010-08-03 Thread Eric Melski
Chiheng Xu wrote: What I want is transparent "parallel make". Make can issue multiple shells simultaneously, but print their outputs in the same order as in a serial make. ElectricAccelerator is a gmake replacement that does exactly this. I wrote about this feature a while ba

Re: Fwd: [RFC]serialize the output of parallel make?

2010-08-03 Thread Eric Melski
Chiheng Xu wrote: What I want is transparent "parallel make". Make can issue multiple shells simultaneously, but print their outputs in the same order as in a serial make. ElectricAccelerator is a gmake replacement that does exactly this. I wrote about this feature a while ba

  1   2   >