Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> From: Paul Smith > CC: David Boyce , bug-make@gnu.org > Date: Fri, 15 Apr 2011 14:53:52 -0400 > > On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote: > > What about the other issue: with the fact that output from a recipe is > > only shown when the entire recipe is done. I think this is a

Re: patch to support output synchronization under -j

2011-04-15 Thread Tim Murphy
I wish I could play with one of those Connection machines and find out about the bottlenecks at j=0.25*cpus that I can see on some machines :-( Regards, Tim On 15 April 2011 19:32, Howard Chu wrote: > Eli Zaretskii wrote: >>> >>> Date: Fri, 15 Apr 2011 10:37:13 -0400 >>> From: David Boyce >>> i

Re: patch to support output synchronization under -j

2011-04-15 Thread Paul Smith
On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote: > What about the other issue: with the fact that output from a recipe is > only shown when the entire recipe is done. I think this is a serious > drawback, at least in some use cases. Can anything be done about > that? I don't see how. The

Re: patch to support output synchronization under -j

2011-04-15 Thread Howard Chu
Eli Zaretskii wrote: Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David Boyce it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. What about people who use "make -j" without limits? They're idiots. I've never

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 1:03 PM, Eli Zaretskii wrote: > Can Make be invoked with its stdout closed by the parent process?  If > it is, will this still work? I've tried to code it such that if anything goes wrong setting up the sync, of which a closed stdout would be one example, it prints a messa

Re: patch to support output synchronization under -j

2011-04-15 Thread Tim Murphy
> What about the other issue: with the fact that output from a recipe is > only shown when the entire recipe is done. I think this is a serious > drawback, at least in some use cases. Can anything be done about > that? The only thing that I can imagine is that one would allow one recipe to have

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> Date: Fri, 15 Apr 2011 10:12:19 -0400 > From: David Boyce > Cc: Eli Zaretskii , bug-make@gnu.org > > Attached is an updated patch with some improvements: Thanks. > - It uses stdout as its semaphore. Can Make be invoked with its stdout closed by the parent process? If it is, will this still

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> Date: Fri, 15 Apr 2011 12:39:56 -0400 > From: David Boyce > Cc: psm...@gnu.org, bug-make@gnu.org > > On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii wrote: > > But this new option uses up 2 additional files per job, doesn't it? > > One or two, as discussed elsewhere in this thread. > > > Eac

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii wrote: > But this new option uses up 2 additional files per job, doesn't it? One or two, as discussed elsewhere in this thread. > Each sub-Make inherits all the file descriptors of all its parents, > grandparents, etc.  If a sub-Make was launched w

Re: patch to support output synchronization under -j

2011-04-15 Thread Paul Smith
On Fri, 2011-04-15 at 18:09 +0300, Eli Zaretskii wrote: > > But yes, any feature > > which consumes additional resources increases the risk of running out > > of that resource. I don't know that recursion comes into it though, > > Each sub-Make inherits all the file descriptors of all its parents,

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> Date: Fri, 15 Apr 2011 10:37:13 -0400 > From: David Boyce > Cc: psm...@gnu.org, bug-make@gnu.org > > > Finally, wouldn't it be a potential problem top inherit so many > > handles to subordinate processes (2 for each running job)?  We could > > run out of available handles in deeply recursive jo

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 9:48 AM, Eli Zaretskii wrote: > So effectively, whenever a job finishes, its parent Make takes the > semaphore, outputs all of the output of that job to the screen, then > releases the semaphore, is that right? Yes, exactly. > And how do you communicate the name of the se

Re: patch to support output synchronization under -j

2011-04-15 Thread Stefano Lattarini
Hello everybody. On Friday 15 April 2011, Eli Zaretskii wrote: > Btw, there will be other side effects, at least on non-Posix > platforms, due to the fact that stuff that was supposed to go to the > screen is redirected to a file instead. Some programs sense that and > behave differently, e.g. wi

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Thu, Apr 14, 2011 at 4:30 PM, David Boyce wrote: > I don't know why this hasn't occurred to me or the authors of similar > programs before, but it appears to be possible to get a lock on any > writable file descriptor - for instance stdout or stderr, or one of > the jobserver-fds. I just change

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> Date: Fri, 15 Apr 2011 12:43:53 +0100 > From: Tim Murphy > Cc: David Boyce , bug-make@gnu.org > > I think it's an inevitable consequence that if you have a long-running > task then the output from it won't appear until it has completely > finished and you won't be able to watch the progress - i

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> Date: Fri, 15 Apr 2011 08:31:43 -0400 > From: David Boyce > Cc: psm...@gnu.org, bug-make@gnu.org > > and I've attached output logs "without.txt" and "with.txt", both using > -j for unlimited jobs. You'll notice that in without.txt the output of > recipes is separated; all the BEGINNING lines sh

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 6:45 AM, Eli Zaretskii wrote: > I lack the higher-level picture.  Can you describe in a few words how > will this work in a running Make?  I don't mean the details of how > files are locked and unlocked (I understand that part), I mean the > larger picture, like how it will

Re: patch to support output synchronization under -j

2011-04-15 Thread Paul Smith
On Fri, 2011-04-15 at 07:18 +0100, Tim Murphy wrote: > Right, of course :-) That is the obvious reason for using files, > thanks! I was biased against writing things to disc which is probably > silly, because it's probably not slow when compared to the size of the > job that's running. I think y

Re: patch to support output synchronization under -j

2011-04-15 Thread Tim Murphy
I replied to this without using "reply all" and I apologise. :-) The order of the output from separate recipes may be anything - depends on how make schedules them but at least the output from any particular recipe will be grouped together and not mixed with the output from others, hence allowing

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> Date: Thu, 14 Apr 2011 16:30:42 -0400 > From: David Boyce > Cc: Eli Zaretskii , bug-make@gnu.org > > I don't know why this hasn't occurred to me or the authors of similar > programs before, but it appears to be possible to get a lock on any > writable file descriptor - for instance stdout or st

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
> Date: Thu, 14 Apr 2011 15:15:15 -0400 > From: David Boyce > Cc: psm...@gnu.org, bug-make@gnu.org > > On Thu, Apr 14, 2011 at 3:00 PM, Eli Zaretskii wrote: > > Yes, but a few words about how is this semaphore supposed to get job > > done, and in fact what kind of "synchronization" will this bri