A bug with libpanel_status.so.0

[c]

Hi there,
When I compile my program, I get this message:
firestarter: error while loading shared 
libraries: libpanel_status.so.0: cannot open shared object file: No such
file or directory
How can I solve this problem knowing that I downloaded
libpanel_status.so.0?
Big thanks.



___
Bug-make mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-make

Documentation bugs?

[C Dasgupta]

This is the manual for GNU make 3.77, May 1998

1) On P 79 (Sec 9.7), looks like '--jobs=[jobs]' should have been
'--jobs[=jobs]'

(I sent this bug also to [EMAIL PROTECTED] since I was not sure where to send
it).

2) By P 43 ( sec 5.7), there have been at least two references to '+'
characters
at the start of a command line. But the feature itself (of what + does)
does
not seem to have been described anywhere yet.

--CB

Crashing when using make with option -j.

[Thiago C. Santini]

We have a computer in our lab we use to run simulations and sometimes
compiling. It's been working just fine for over 6 months now but during the
last week we had it's first 3 crashes(only fixable by hard restarting the
machine). I'm not a very experienced linux user so all I tried to do to find
out the crash reason was looking into /var/logs without any sucess. So I
tracked down which programs were being used when the crash happened and
tried to reproduce the crash.
So one of the targets was make which I was running using $make -j 32. After
a few runs trying to make the computer crash again I was able to make it
crash using $make -j 128
I'm not sure wheter the reason for the crash is make or gcc but I do know it
crashes using that command, so I'd like to know if anyone has ever had this
problem before and how I can get more information about the crash that could
lead me to the exact crash reason.
Everything described here happened over ssh though I don't think that does
make a difference.

Machine configuration when the crash happened:
Core i7 @ 2.67GHZ
3GB RAM
$uname -a
Linux groelandia 2.6.28-16-generic #57-Ubuntu SMP Wed Nov 11 09:47:24 UTC
2009 i686 GNU/Linux
Ubuntu version is 9.04(32 bits) (yeah we do also have a 64 bits version
installed =p)

If any info is needed or I sent this message to the wrong place just let me
know.
Thanks in advance,
-- 
Thiago C. Santini
ECP/07
___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Crashing when using make with option -j.

[Thiago C. Santini]

On Thu, Mar 18, 2010 at 5:41 PM, Paul Smith  wrote:

> On Thu, 2010-03-18 at 14:57 -0300, Thiago C. Santini wrote:
> > We have a computer in our lab we use to run simulations and sometimes
> > compiling. It's been working just fine for over 6 months now but
> > during the last week we had it's first 3 crashes(only fixable by hard
> > restarting the machine). I'm not a very experienced linux user so all
> > I tried to do to find out the crash reason was looking into /var/logs
> > without any sucess. So I tracked down which programs were being used
> > when the crash happened and tried to reproduce the crash.
> > So one of the targets was make which I was running using $make -j 32.
> > After a few runs trying to make the computer crash again I was able to
> > make it crash using $make -j 128
> > I'm not sure wheter the reason for the crash is make or gcc but I do
> > know it crashes using that command, so I'd like to know if anyone has
> > ever had this problem before and how I can get more information about
> > the crash that could lead me to the exact crash reason.
> > Everything described here happened over ssh though I don't think that
> > does make a difference.
>
> You are running on a system with (from what I can tell) 8 processing
> threads.  So, why are you trying to run at -j32?  That's 4 jobs per
> processor, plus all the overhead for I/O which is shared between them.
>
> I suspect that when you say "crashed", that the system didn't really
> crash.  It probably got so bogged down and busy thrashing swap etc. that
> it became unresponsive.  If the system really did panic then this is a
> bug in the kernel (if anywhere), or else maybe you're starting to have a
> hardware issue (bad RAM or similar).
>
> In any case, this is not a bug in either make or GCC.  If starting that
> many jobs causes your system to become unresponsive, then my
> recommendation is to not do that :-).  Your system is apparently not
> powerful enough to support that level of parallelism.
>
>
> Yeah, that was my first thought when using -j, 8 processors each one with
hyper-threading should be optimized with 16 jobs but when testing it I got
better results with 32 jobs and that was working just fine till last week,
so I just sticked to it =p

Is there anyway to find out if it really was a kernel panic? Like a log or
something? Any link to good tutorials about debugging a crash are welcomed
aswell :-) I guessed it was a kernel crash because even the video signal was
killed once(the screen wouldn't show anything at all).

The real problem is that the machine crashed when there was more than one
person working over ssh and I thought maybe someone was compiling at the
same time as me and that made too many jobs.

Anyway, thanks for the help,
-- 
Thiago C. Santini
ECP/07
___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Crashing when using make with option -j.

[Thiago C. Santini]

On Fri, Mar 19, 2010 at 4:39 AM, Eli Zaretskii  wrote:

> > Date: Thu, 18 Mar 2010 16:26:19 -0700
> > From: tom honermann 
> > Cc: bug-make@gnu.org
> >
> >
> > On 3/18/2010 2:22 PM, Thiago C. Santini wrote:
> > > Yeah, that was my first thought when using -j, 8 processors each one
> > > with hyper-threading should be optimized with 16 jobs but when testing
> > > it I got better results with 32 jobs and that was working just fine
> > > till last week, so I just sticked to it =p
> > >
> > > Is there anyway to find out if it really was a kernel panic? Like a
> > > log or something? Any link to good tutorials about debugging a crash
> > > are welcomed aswell :-) I guessed it was a kernel crash because even
> > > the video signal was killed once(the screen wouldn't show anything at
> > > all).
> > >
> > > The real problem is that the machine crashed when there was more than
> > > one person working over ssh and I thought maybe someone was compiling
> > > at the same time as me and that made too many jobs.
> > >
> > You may be falling victim to the kernel OOM killer
> > (http://linux-mm.org/OOM_Killer)
>
> If processes are killed by OOM Killer, there's a prominent message to
> that effect in the system logs (/var/log/messages or some such).
> Search for "oom" (IIRC) in the log.
>

No such thing on any log. Grepped for both "oom" and "IIRC".
No new crashes happened after I started using -j 16, and since I don't have
time nor experience to debug this I'll leave it alone for a week or two and
start getting to know the kernel better and then try debugging it again.

Thanks for the help.
-- 
Thiago C. Santini
ECP/07
___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

improving the -l option

[Sven C. Dack]

Hello,

I have created a patch to improve make's behaviour with regards to parallel 
execution and system load.


On Linux do the load averages refresh only every 5 seconds. Further do load 
averages only ever represent past but not present load. As a consequence does 
this lead to idle times during parallel builds where make is waiting for the 
load averages to come down. It can also spawn excessive amounts of processes 
(although it tries to balance it with its own heuristic).


A better way to achieve a stable load is to use the active number of running 
processes & threads over of the load averages. This number can be found in 
/proc/loadavg on Linux and is updated in real-time. An example:


$ cat /proc/loadavg
0.00 2.72 6.89 1/404 1890

Here it shows 1 (out of 404) running processes. The patch allows make to use the 
4th value from /proc/loadavg.


The overall improvement for a parallel build on an 8-core CPU:

-j -l 24 (old): 269.993128122 seconds time elapsed
-j -l 24 (new): 231.300406406 seconds time elapsed

In comparison:

-j 24: 230.482726977 seconds time elapsed

The behaviour of the -l  option with the patch is now much closer to that 
of the -j  option and shows almost identical build times for an empty system.


The test suite passes successfully with the patch.

You'll find two patch files against make-4.2.1 in the attachment. One for 
configure.ac (includes a test for /proc/loadavg usability) and the other for 
job.c (modifies the function load_too_high()).


Regards,
Sven

--- make-4.2.1/configure.ac 2016-06-06 13:27:31.0 +0100
+++ make-4.2.1.1/configure.ac   2016-10-09 18:33:34.574257275 +0100
@@ -176,10 +176,30 @@
 AS_IF([test "$have_guile" = yes],
   [AC_DEFINE([HAVE_GUILE], [1], [Embed GNU Guile support])])
 
 AM_CONDITIONAL([HAVE_GUILE], [test "$have_guile" = yes])
 
+AC_CHECK_FUNCS[open close read lseek atoi]
+AC_CACHE_CHECK([for /proc/loadavg usability], [ac_cv_proc_loadavg],
+  [ac_cv_proc_loadavg=no
+   AC_RUN_IFELSE([AC_LANG_SOURCE([[#include 
+#include 
+#include 
+int main() {
+  int fd, i, n; char str[64];
+  fd = open("/proc/loadavg", O_RDONLY);
+  if (fd < 0 || read(fd, str, 64) < 0 || lseek(fd, 0, SEEK_SET) < 0) return 1;
+  for (str[63] = i = n = 0; i < 3 && n < 64; ++n) if (str[n] == ' ') ++i;
+  return i < 3 || atoi(&str[n]) < 1 || close(fd) < 0;}]])],
+  [ac_cv_proc_loadavg=yes],
+  [ac_cv_proc_loadavg=no],
+  [ac_cv_proc_loadavg="no (cross-compiling)"])])
+AS_IF([test "$ac_cv_proc_loadavg" = yes],
+[ AC_DEFINE([HAVE_PROC_LOADAVG], [1],
+[Define to 1 when /proc/loadavg exists and appears usable])
+])
+
 AC_FUNC_GETLOADAVG
 
 # AC_FUNC_GETLOADAVG is documented to set the NLIST_STRUCT value, but it
 # doesn't.  So, we will.
 
--- make-4.2.1/job.c2016-05-21 21:22:32.0 +0100
+++ make-4.2.1.1/job.c  2016-10-09 19:12:07.425523927 +0100
@@ -1925,10 +1925,60 @@
 static int
 load_too_high (void)
 {
 #if defined(__MSDOS__) || defined(VMS) || defined(_AMIGA) || 
defined(__riscos__)
   return 1;
+#elif defined(HAVE_PROC_LOADAVG)
+  /* Read from /proc/loadavg and use the number of active running
+ tasks & threads to determine if we can start another process. */
+#define PROC_LOADAVG_SIZE 64
+  static int fd = -2;
+  char proc_loadavg[PROC_LOADAVG_SIZE];
+  int i, p;
+
+  if (max_load_average < 0)
+return 0;
+
+  if (fd == -2)
+{
+  fd = open ("/proc/loadavg", O_RDONLY);
+  if (fd < 0)
+   {
+ perror_with_name (_("cannot open /proc/loadavg: "), "open");
+ fd = -1;
+ return 0;
+   }
+}
+  if (read (fd, proc_loadavg, PROC_LOADAVG_SIZE) < 0)
+{
+  perror_with_name (_("cannot read /proc/loadavg: "), "read");
+  close(fd);
+  fd = -1;
+  return 0;
+}
+  if (lseek (fd, 0, SEEK_SET) < 0)
+{
+  perror_with_name (_("cannot seek on /proc/loadavg: "), "lseek");
+  close(fd);
+  fd = -1;
+  return 0;
+}
+  proc_loadavg[PROC_LOADAVG_SIZE-1] = '\0';
+  for (i = p = 0; i < 3 && p < PROC_LOADAVG_SIZE; ++p)
+{
+  if (proc_loadavg[p] == ' ')
+   ++i;
+}
+  if (i < 3)
+{
+  errno = ENOTSUP;
+  perror_with_name (_("unsupported format of /proc/loadavg: "), 
"load_too_high");
+  close (fd);
+  fd = -1;
+  return 0;
+}
+  return max_load_average < (double) atoi (&proc_loadavg[p]);
 #else
   static double last_sec;
   static time_t last_now;
   double load, guess;
   time_t now;
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

improving the -l option (addendum)

[Sven C. Dack]

A minor correction. I've missed a test for an error case. Updated patch can be 
found in the attachments.


Sven

--- make-4.2.1/configure.ac 2016-06-06 13:27:31.0 +0100
+++ make-4.2.1.1/configure.ac   2016-10-09 18:33:34.574257275 +0100
@@ -176,10 +176,30 @@
 AS_IF([test "$have_guile" = yes],
   [AC_DEFINE([HAVE_GUILE], [1], [Embed GNU Guile support])])
 
 AM_CONDITIONAL([HAVE_GUILE], [test "$have_guile" = yes])
 
+AC_CHECK_FUNCS[open close read lseek atoi]
+AC_CACHE_CHECK([for /proc/loadavg usability], [ac_cv_proc_loadavg],
+  [ac_cv_proc_loadavg=no
+   AC_RUN_IFELSE([AC_LANG_SOURCE([[#include 
+#include 
+#include 
+int main() {
+  int fd, i, n; char str[64];
+  fd = open("/proc/loadavg", O_RDONLY);
+  if (fd < 0 || read(fd, str, 64) < 0 || lseek(fd, 0, SEEK_SET) < 0) return 1;
+  for (str[63] = i = n = 0; i < 3 && n < 64; ++n) if (str[n] == ' ') ++i;
+  return i < 3 || atoi(&str[n]) < 1 || close(fd) < 0;}]])],
+  [ac_cv_proc_loadavg=yes],
+  [ac_cv_proc_loadavg=no],
+  [ac_cv_proc_loadavg="no (cross-compiling)"])])
+AS_IF([test "$ac_cv_proc_loadavg" = yes],
+[ AC_DEFINE([HAVE_PROC_LOADAVG], [1],
+[Define to 1 when /proc/loadavg exists and appears usable])
+])
+
 AC_FUNC_GETLOADAVG
 
 # AC_FUNC_GETLOADAVG is documented to set the NLIST_STRUCT value, but it
 # doesn't.  So, we will.
 
--- make-4.2.1/job.c2016-05-21 21:22:32.0 +0100
+++ make-4.2.1.1/job.c  2016-10-09 20:02:43.047767201 +0100
@@ -1925,10 +1925,60 @@
 static int
 load_too_high (void)
 {
 #if defined(__MSDOS__) || defined(VMS) || defined(_AMIGA) || 
defined(__riscos__)
   return 1;
+#elif defined(HAVE_PROC_LOADAVG)
+  /* Read from /proc/loadavg and use the number of active running
+ tasks & threads to determine if we can start another process. */
+#define PROC_LOADAVG_SIZE 64
+  static int fd = -2;
+  char proc_loadavg[PROC_LOADAVG_SIZE];
+  int i, p;
+
+  if (max_load_average < 0 || fd == -1)
+return 0;
+
+  if (fd == -2)
+{
+  fd = open ("/proc/loadavg", O_RDONLY);
+  if (fd < 0)
+   {
+ perror_with_name (_("cannot open /proc/loadavg: "), "open");
+ fd = -1;
+ return 0;
+   }
+}
+  if (read (fd, proc_loadavg, PROC_LOADAVG_SIZE) < 0)
+{
+  perror_with_name (_("cannot read /proc/loadavg: "), "read");
+  close(fd);
+  fd = -1;
+  return 0;
+}
+  if (lseek (fd, 0, SEEK_SET) < 0)
+{
+  perror_with_name (_("cannot seek on /proc/loadavg: "), "lseek");
+  close(fd);
+  fd = -1;
+  return 0;
+}
+  proc_loadavg[PROC_LOADAVG_SIZE-1] = '\0';
+  for (i = p = 0; i < 3 && p < PROC_LOADAVG_SIZE; ++p)
+{
+  if (proc_loadavg[p] == ' ')
+   ++i;
+}
+  if (i < 3)
+{
+  errno = ENOTSUP;
+  perror_with_name (_("unsupported format of /proc/loadavg: "), 
"load_too_high");
+  close (fd);
+  fd = -1;
+  return 0;
+}
+  return max_load_average < (double) atoi (&proc_loadavg[p]);
 #else
   static double last_sec;
   static time_t last_now;
   double load, guess;
   time_t now;
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Recursive GMAKE with -j option

[Bhatt, Milan C]

Hello,
 
Has anyone run into 
problems when performing recursive GMAKE AND specifying the -j option?  
I've noticed that there are no problems with the following:
 
gmake 
-j
 
when you're doing 
recursive gmake.  But I run into problems when I specify an integer value 
for the -j option:
 
gmake -j 
5
 
I get the following 
message:
    

    
> warning: jobserver unavailable: using -j1. Add `+' to parent make 
rule
 
Anyone run into this 
problem?  Any possible solutions to this?
 
Thanks,
Milan

RE: Recursive GMAKE with -j option

[Bhatt, Milan C]

I just figured out the issue I was having with doing recursive gmake and
specifying the -jN option at the same time.  I was using tcsh by setting
$(SHELL) variable, and it seems tcsh isn't POSIX compliant.  Linking that
information with following entry within the gmake history document gave me
the solution:

>* A "job server" feature, suggested by Howard Chu <[EMAIL PROTECTED]>.
>  On systems that support POSIX pipe(2) semantics, GNU make can now pass
>  -jN options to submakes rather than forcing them all to use -j1.  The
>  top make and all its sub-make processes use a pipe to communicate with
>  each other to ensure that no more than N jobs are started across all
>  makes.  To get the old behavior of -j back, you can configure make
>  with the --disable-job-server option.

Because tcsh wasn't fully POSIX compliant, it could not properly establish a
pipe for communication between the sub-make processes leading to gmake
complaining that the "job sever" wasn't available.

Now the question is which shell would be best to use?  I'd preferrably like
to use bash because of the additional features it has over sh, but I'm not
too sure about how POSIX compliant it is.

Thanks,
Milan

-Original Message-
From: Paul D. Smith [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 29, 2001 12:06 PM
To: Bhatt, Milan C
Cc: '[EMAIL PROTECTED]'
Subject: Re: Recursive GMAKE with -j option 


%% "Bhatt, Milan C" <[EMAIL PROTECTED]> writes:
 
  bmc> I get the following message:

  bmc> warning: jobserver unavailable: using -j1. Add `+' to parent make
rule

See the GNU make manual section on parallel builds and the section
on error messages for a description of what this means.

If that doesn't answer your questions, re-post with appropriate
information: what version of GNU make you're using, what host/OS/version
you're using it on, and show an example of the rule you use to invoke a
sub-make.

-- 

---
 Paul D. Smith <[EMAIL PROTECTED]>  Find some GNU make tips at:
 http://www.gnu.org  http://www.paulandlesley.org/gmake/
 "Please remain calm...I may be mad, but I am a professional." --Mad
Scientist


___
Bug-make mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-make

Bug with semicolon in target specific variable?

[Bhatt, Milan C]

Hello,
 
It seems GMAKE doesn't like 
having semicolons within the value of a target specific 
variable.
 
 
a : SOME_VAR := c; 
d
a :
   @echo 
$(SOME_VAR)
 
The above exampe gives me 
the following error:
 
c/bin/sh: d: Execute 
permission denied.gmake: *** [a] Error 127
 
So it prints 'c' as 
expected and then tries to execute 'd' as if it were a command.  As a 
workaround I used quotes around the entire value of the target specific variable 
and everything seemed to work fine until I hit about 130 or more 
characters:
 
 
a : SOME_VAR := "c; d; 
e"    
# 'e' x 130+ characters
a :
   @echo 
$(SOME_VAR)
 
 
Once there are more than a 
130 characters  AND there is a semicolon within the quoted string, I get 
this error message:
 
Makefile:3: Malformed 
per-target variable definitionBus error (core dumped)
 
 
This error disappears once 
I remove the semicolon from the variable.  Any ideas?
 
Thanks,
Milan

RE: Bug with semicolon in target specific variable?

[Bhatt, Milan C]

Yeah I did figure that semicolons had special meaning in GMAKE, but my first
example is actually a springboard for the second one.  If I quote everything
with "", then it works except for this strange character limit.

FYI, the \ also has the special meaning of line continuation in GMAKE ;)

Milan

-Original Message-
From: Johan Bezem [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 17, 2002 5:05 AM
To: Bhatt Milan C
Cc: '[EMAIL PROTECTED]'
Subject: Re: Bug with semicolon in target specific variable?


Semicolons have a special meaning in GNU-make. RTFM!
Escaping with \; might help. Didn't try, though.

Johan Bezem
CSK Software AG

> "Bhatt, Milan C" wrote:
> 
> Hello,
> 
> It seems GMAKE doesn't like having semicolons within the value of a
> target specific variable.
> 
> 
> a : SOME_VAR := c; d
> a :
>@echo $(SOME_VAR)
> 
> The above exampe gives me the following error:
> 
> c
> /bin/sh: d: Execute permission denied.
> gmake: *** [a] Error 127
> 
> So it prints 'c' as expected and then tries to execute 'd' as if it
> were a command.  As a workaround I used quotes around the entire value
> of the target specific variable and everything seemed to work fine
> until I hit about 130 or more characters:
> 
> 
> a : SOME_VAR := "c; d;
> e"
> # 'e' x 130+ characters
> a :
>@echo $(SOME_VAR)
> 
> 
> Once there are more than a 130 characters  AND there is a semicolon
> within the quoted string, I get this error message:
> 
> Makefile:3: Malformed per-target variable definition
> Bus error (core dumped)
> 
> 
> This error disappears once I remove the semicolon from the variable.
> Any ideas?
> 
> Thanks,
> Milan

___
Bug-make mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-make

implicit rule invocation issue

[Hakenson, Gary C.]

Hi developers,

I have an issue with make that I am unable to resolve on my own.

My makefile contains an implicit rule as follows:

%.d : %.c
(command to create a .d file from the .c file here)

I also have the VPATH variable set as follows:

VPATH = /d

When I run make, it finds both the target (.d) and the prerequisite (.c) in
the /d directory and since the target is newer then the prerequisite nothing
is done. This is as expected since the current directory does not contain a
.d file.

There exists in the current directory a .c file. What I want to happen is
for the above implicit rule to be invoked on this .c file to produce a .d
file (in the current directory) but this is not happening.

Section 10.5.4 How Patterns Match of the GNU Make book states: "The
directories are ignored only for the purpose of finding an implicit rule to
use, not in the application of that rule.". This explains the behavior I am
seeing.

Not invoking the above implicit rule for the target and prerequisite in the
/d directory is understandable. However, I need the rule to be invoked on a
prerequisite in the current directory to produce the target (which does not
exist) in the current directory.

Do you have any suggestions as to how this may be accomplished?

Thanks for your attention.

Gary Hakenson
505-678-2676




___
Bug-make mailing list
[EMAIL PROTECTED]
http://lists.gnu.org/mailman/listinfo/bug-make

PATCH: make-3.81

[Michael C Thompson]

Hi,

In file implicit.c, there seems to be a variable 'remove_explicit_deps' 
which, as far as I can tell, serves as a flag to "remove all the 
dependencies that didn't come from this implicit rule", as per the 
comment in the code.


However, this variable is initialized to 0, then set to 0, then checked 
in an if statement, which implies this value is a compile time only 
flag. Since this code can never be reached, it seems logical to remove 
it; (or if it should exist, at least make it a #define.


Below is a patch which removes this "dead code", which was found by 
Coverity.


Thanks,
Mike

---

--- make-3.81/implicit.c2006-04-01 00:36:40.0 -0600
+++ make-3.81-diff/implicit.c   2006-04-26 09:23:23.801794184 -0500
@@ -219,9 +219,6 @@
  (see struct idep for details). */
   struct idep* deps = 0;

-  /* 1 if we need to remove explicit prerequisites, 0 otherwise. */
-  unsigned int remove_explicit_deps = 0;
-
   /* Names of possible dependencies are constructed in this buffer.  */
   register char *depname = (char *) alloca (namelen + 
max_pattern_dep_length);


@@ -445,8 +442,6 @@

  rule = tryrules[i];

-  remove_explicit_deps = 0;
-
  /* RULE is nil when we discover that a rule,
 already placed in TRYRULES, should not be applied.  */
  if (rule == 0)
@@ -802,21 +797,6 @@
  This includes the intermediate files, if any.
  Convert them into entries on the deps-chain of FILE.  */

-  if (remove_explicit_deps)
-{
-  /* Remove all the dependencies that didn't come from
- this implicit rule. */
-
-  dep = file->deps;
-  while (dep != 0)
-{
-  struct dep *next = dep->next;
-  free_dep (dep);
-  dep = next;
-}
-  file->deps = 0;
-  }
-
   expl_d = file->deps; /* We will add them at the end. */
   d_ptr = &file->deps;





___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

PATCH: make-3.81 - read.c bug?

[Michael C Thompson]

Hi,

In file read.c, the function multi_glob() has #ifndef NO_ARCHIVES code 
which I believe has a bug.


Once you enter the large encompassing for loop memname get sets to 
either NULL or non-NULL (due to a malloc). Then, entering case 0, i gets 
set to gl.gl_patchc, and a while loop churns over i until it reaches 0. 
During this churning, memname is checked to be non-NULL, if so, then its 
 derefernced, otherwise, it is not. However, during this while loop, 
memname is unconditionally free'd. As far as I can tell, if i > 1, then 
this will dereference free'd memory.


I believe this is a bug, although I might be missing some magic 
somewhere. Regardless, below is a patch which addresses this ""use after 
free" issue, which was found by Coverity. The solution is merely to set 
memname to be NULL after the free, for lack of better understanding of 
the purpose of this code.


Thanks,
Mike

---

--- make-3.81/read.c2006-03-17 08:24:20.0 -0600
+++ make-3.81-diff/read.c   2006-04-26 10:06:31.605388240 -0500
@@ -3104,6 +3104,7 @@
  }

free (memname);
+   memname = 0;
  }
else
 #endif /* !NO_ARCHIVES */




___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

[Sven C. Dack]

Hello,

this isn't a bug, but it's how the -j option works.

From the manual:

   -j [jobs], --jobs[=jobs]
Specifies the number of jobs (commands) to run 
simultaneously.  If
there is more than one -j option, the last one is 
effective.   If
the  -j  option  is given without an argument, make will 
not limit

the number of jobs that can run simultaneously.

Specifying -j1 or -j 1 are the same here.

Your problem is that you are using numbers as target names. So it's not 
a bug, because you are literally telling make to run only a single job.


You either have to restrict the number of jobs by giving an explicit 
count or by limiting it with a load average ( -l option) or use 
non-numerical make targets such as "t1 t2 t3 ..." or simply add another 
flag after -j to make.


From my own experience is it generally better to limit the number of 
jobs to what is reasonable or otherwise risk setting off a fork bomb - a 
run-away process which spawns too many process too fast and brings the 
system to a halt.


If it really needs to be unlimited then using non-numerical targets is 
what you can do:


$ seq 1000 | sed 's,^,t,' | xargs -n1000 make -j

which results in:

$ make -j t1 t2 t3 ...

Or if it needs to be unlimited and it has to be numerical arguments can 
you simply do this:


$ seq 1000 | xargs -n1000 make -j -C .

which results in:

$ make -j -C . 1 2 3 ...

The -C option tells make to change the directory to the current 
directory and avoids the target 1 to be treated as an optional argument 
to -j.


Cheers



On 21/06/17 20:39, Michael Builov wrote:

Follow-up Comment #1, bug #49844 (project make):

Hello.

There is a bug in your trivial example:

1) 'seq 1000 | xargs -n1000 make -j5'

expands to

'make -j5 1 2 3 4 5 6 7 ...'

and

2) 'seq 1000 | xargs -n1000 make -j'

expands to

'make -j 1 2 3 4 5 6 7 ...'

As we can see, in second example '1' after '-j' make interprets as number of
jobs, so "it not do any parallelization at all".


 ___

Reply to this item at:

   <http://savannah.gnu.org/bugs/?49844>

___
   Message sent via/by Savannah
   http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make




___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

[Sven C. Dack]

Hello Paul,

I was thinking of "--", but didn't test it, because it isn't being 
mentioned in the man page nor in the info page.


Cheers



On 22/06/17 13:10, Paul Smith wrote:

On Thu, 2017-06-22 at 13:01 +0100, Sven C. Dack wrote:

You either have to restrict the number of jobs by giving an explicit
count or by limiting it with a load average ( -l option) or use
non-numerical make targets such as "t1 t2 t3 ..." or simply add another
flag after -j to make.
$ seq 1000 | xargs -n1000 make -j -C .

which results in:

$ make -j -C . 1 2 3 ...

GNU make, as with all GNU tools (and with all well-formed POSIX
commands) accepts the "--" option to mean "everything after this is not
an option even if it looks like one".

So, the simplest solution is to use:

   seq 1000 | xargs -n1000 make j --

I don't really understand why you use the pipe to xargs.  Wouldn't it be
simpler to just say:

   make -j -- $(seq 1000)

?

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Baffled by cc -o in Make manual

[Sven C. Dack]

Hello,

this is completely normal.

The compiler calls the assembler and linker automatically when needed 
and depending on the input files. One doesn't actually call the 
assembler and linker manually in most cases. You can do:


    cc -c prog.c -o prog.o

This will compile the file 'prog.c' into assembly code, pass it to the 
assembler, which then produces the 'prog.o' file. You can also do:


cc -S prog.c -o prog.s

This tells the compiler to only produce assembly output and to write it 
to 'prog.s'. When you look at it can you see the assembler instructions, 
which would otherwise have been passed onto the assembler for it to 
produce an object file.


The compiler simply acts as a front-end to the assembler and linker. You 
can let the compiler do it all in one go where it calls the assembler 
and linker to build the final program:


cc prog.c -o prog

Because programs can become quite large and managing the source requires 
many files is it more convenient to split the task into multiple files:


    cc -c prog1.c -o prog1.o

    cc -c prog2.c -o prog2.o

cc -c prog3.c -o prog3.o

cc -o prog prog1.o prog2.o prog3.o

And let's not forget the libraries. These also are being handled through 
the compiler even when it's the linker, which does the actual work:


cc prog1.o prog2.o prog3.o mylib.a -lm -lc -o prog

And so on. This concept lets you manage the development of a program in 
convenient chunks and doesn't force you to keep an eye on the whole code 
all the time.


This is then why it has 'make' and Makefiles. It allows you to define 
how a program is being put together, how to call the compiler, which 
files belong together, and so on. And because 'make' can detect when 
files have changed does it make this particularly easy, because only 
what has changed then needs to be recompiled and thereby saves you time.


A simple, first Makefile for the three files 'prog1.c', 'prog2.c' and 
'prog3.c' could look like this:


prog1.o: prog1.c

    cc prog1.c -o prog1.o

prog2.o: prog2.c

cc prog2.c -o prog2.o

prog3.o: prog3.c

cc prog3.c -o prog3.o

prog: prog1.o prog2.o prog3.o

cc prog1.o prog2.o prog3.o -o prog

When you now type 'make prog' will it check if a file by that name 
exists and if it's newer than 'prog1.o', 'prog2.o' and 'prog3.o'. If so 
then it knows 'prog' is up to date and doesn't need to be remade. 
Otherwise will it invoke "cc prog1.o prog2.o prog3.o -o prog" to produce 
a newer version.


But it doesn't stop there. Because it also has rules to make 'prog1.o' 
and the other two will it first check if perhaps these are out of date 
and if these need to be remade.


Thus, when one of the files 'prog1.c', 'prog2.c' or 'prog3.c' have 
changed and become newer will 'make' only remake those pieces of your 
program, which really require to be remade.


Once this is understood does everything else about 'make' fall into 
place. It's all about breaking large work into manageable chunks and to 
avoid extra work be defining clever rules, which will do all the 
necessary steps for you in order to put it all together.


I hope this helps you to understand the use of 'make' and 'cc'.

Cheers,

Sven



On 01/08/17 17:32, Ray Foulkes wrote:


Hello, like many reports I suppose I have this totally wrong but the 
very first example on page 4 of the make manual at


https://www.gnu.org/software/make/manual/make.pdf reads:


edit : main.o kbd.o command.o display.o \
insert.o search.o files.o utils.o
cc -o edit main.o kbd.o command.o display.o \
insert.o search.o files.o utils.o


I am totally baffled as to what cc -o is doing there. Surely it should 
be ld?? and then the -o is for "output" i.e. create "edit" as an 
executable.


On the other hand it continues on through the other examples of using 
variables in subsequent pages so I am beginning to doubt myself.


I cannot understand how make knows to call ld if it is not explicit 
somewhere in the file.


Please ignore if I am just being stupid... If so, boy, have you got me 
baffled - and I have only got to page 4!



Regards, Ray Foulkes




___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Target-specific variable in subdirectory problem

[Sven C. Dack]

Hello,

try it with the following rules:

test-%: $(eval FOO = BAR)
test-%:
echo $(FOO)

$ make test-stem
echo BAR
BAR
$ make subdir/test-stem
echo BAR
BAR

I cannot tell you why it fails in your example. My guess is that it's 
just not good practise to use a plain assignment for a prerequisite and 
that it's only working for backwards compatibility and not meant to be a 
full feature. I'm not the authority on make and your guess will be as 
good as mine.


Cheers,

Sven


On 02/08/17 11:19, Benjamin Cama wrote:

Hi,

I may be doing something wrong, but the following Makefile gives me
strange results: the target-specific variable does not apply when used
for a target in a subdirectory.

 test-%: FOO = BAR
 test-%:
echo $(FOO)
 
E.g.:


 $ make test-stem
 echo BAR
 BAR
 $ make subdir/test-stem
 echo
 


What am I doing wrong?

PS: please Cc me, I'm not subscribed. Thanks.



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Target-specific variable in subdirectory problem

[Sven C. Dack]

It's my understanding that we don't actually know what Benjamin wants to 
do, because nobody has asked. So I don't see how this would make me wrong.


From his mail am I assuming he wants his Makefile to print "BAR" for a 
target "test-stem" as well as "subdir/test-stem", which is what he will 
get when he uses $(eval ...).


Respectfully,

Sven



On 02/08/17 20:12, Kyle Rose wrote:

On Wed, Aug 2, 2017 at 3:03 PM, Sven C. Dack  wrote:

Hello,

try it with the following rules:

test-%: $(eval FOO = BAR)
test-%:
 echo $(FOO)

FOO is a global variable in this case. Not sure what you're trying to
demonstrate here.

Kyle

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make




___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Target-specific variable in subdirectory problem

[Sven C. Dack]

I suggest you turn your attention towards him and ask him what he wants.

My example does exactly what I say it does. It's just not your place to 
tell me I'm wrong when you don't know what he wants his makefile to do 
and so far haven't responded to him and only to me.


Or am I missing something? We are all adults here, right?



On 02/08/17 20:55, Paul Smith wrote:

On Wed, 2017-08-02 at 20:40 +0100, Sven C. Dack wrote:

It's my understanding that we don't actually know what Benjamin wants to
do, because nobody has asked. So I don't see how this would make me wrong.

  From his mail am I assuming he wants his Makefile to print "BAR" for a
target "test-stem" as well as "subdir/test-stem", which is what he will
get when he uses $(eval ...).

Because he is using a pattern-specific variable assignment instead of
just assigning the variable globally we can infer that he would like to
have the "FOO = BAR" assignment only in the context of targets that
match that pattern but not other targets.

So for this makefile:

   test-%: FOO = BAR
   test-%:
   echo $(FOO)

   other:
   echo $(FOO)

He would like to see:

   $ make test-foo
   echo BAR
   BAR

   $ make other
   echo

If we change the pattern-specific variable to use $(eval FOO = BAR) he
would see:

   $ make test-foo
   echo BAR
   BAR

   $ make other
   echo BAR
   BAR




___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Target-specific variable in subdirectory problem

[Sven C. Dack]

No. You're mistake is to assume to know what he wants from looking at 
his Makefile when you also know it's not working for him. You cannot 
know what it is he wants when he apparently fails to express it in a 
Makefile. So you're still only assuming. You base your assumption on the 
fact that he puts the assignment into a prerequisite. I'm basing mine on 
the assumption that he wants an identical behaviour for two different 
targets.


It doesn't make your guess better (or worse) than mine. It's up to 
Benjamin to tell us. It's not up to you nor me. So let's be fair here.


What I meant by saying that you didn't respond is that you didn't 
respond to his initial e-mail containing his problem, but that you 
hopped onto my response, second-guessed me, told me I'm wrong when 
really it's just an opinion you're having, and yet still haven't given 
any solution to what you think he wants to do. You've basically just 
lectured him about what he cannot have.


That's why I gave him an example. It will help him to decide what it is 
he wants and to be clearer about his intention. Perhaps you're right and 
whatever he wants to do isn't going to work with make, but until then 
it's best to keep an open mind, to stay positive and to try to find 
something that might work for him and not tell each other whose wrong 
and right.


Cheers



On 02/08/17 21:36, Paul Smith wrote:

On Wed, 2017-08-02 at 21:16 +0100, Sven C. Dack wrote:

I suggest you turn your attention towards him and ask him what he
wants.

The example in his question makes very clear what he wants: he wants a
pattern-specific variable assignment.

The solution you provided was not a pattern-specific variable
assignment.  It was an abstracted way of writing a global variable
assignment.


My example does exactly what I say it does. It's just not your place
to tell me I'm wrong when you don't know what he wants his makefile
to do and so far haven't responded to him and only to me.

First, I did respond to him and I explained exactly why his example
wasn't working the way he anticipated.

Second, we don't need to ask him more about what he wants the makefile
to do, because the intent of the question is self-evident.

Finally, I didn't intend any attack.  I'm just explaining how the
example you gave works and why it doesn't meet the OPs request.  It's
not personal.  Indeed in your original message it seemed like you
weren't familiar with pattern-specific variables so I provided more
context.


Or am I missing something? We are all adults here, right?

Is that a trick question?




___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Fwd: Re: Target-specific variable in subdirectory problem

[Sven C. Dack]

Hello Benjamin,

it's a common practise to create a Makefile for each subdirectory and
have it include definitions from a parent Makefile. That's probably why
you don't see that many directory-spanning features in make. It's easier
to manage Makefiles in a structure similar to the source code instead of
having it all packed into one Makefile, especially the larger your
project grows.

Here is an example of what you can. Assume you have a top-level
directory with three subdirectories and separate Makefiles in them:

Makefile  Makefile.subs  sub-a/Makefile  sub-sub/Makefile sub-z/Makefile

--- Makefile ---
SUBS := $(wildcard sub-*/)
SUB_TARGETS := $(patsubst %,%%,$(SUBS))

$(SUB_TARGETS):
make -C $(dir $@) $(notdir $@)

FOO := "Makefiles"
include Makefile.subs

--- Makefile.subs ---
test-%:
@echo "$(FOO)"

--- sub-a/Makefile ---
FOO := "Hello"
include ../Makefile.subs

--- sub-sub/Makefile ---
FOO := "of"
include ../Makefile.subs

--- sub-z/Makefile ---
FOO := "World"
include ../Makefile.subs

When you now enter "make sub-a/test-1 sub-z/test-2 sub-sub/test-3
test-4" will you get:

make -C sub-a/ test-1
make[1]: Entering directory '/home/sven/x/sub-a'
Hello
make[1]: Leaving directory '/home/sven/x/sub-a'
make -C sub-z/ test-2
make[1]: Entering directory '/home/sven/x/sub-z'
World
make[1]: Leaving directory '/home/sven/x/sub-z'
make -C sub-sub/ test-3
make[1]: Entering directory '/home/sven/x/sub-sub'
of
make[1]: Leaving directory '/home/sven/x/sub-sub'
Makefiles


The Makefiles of the subdirectories are simple and straight-forward.
These define FOO and only include a parent Makefile called
Makefile.subs, which defines your "test-%" rule to print the value of FOO.

The top-level Makefile uses the 'wildcard' and 'patsubst' functions of
make to create a list of targets "sub-a/% sub-z/% sub-sub/%" and uses it
for a rule, which will call a second make to run it in a subdirectory.

Note the the top-level Makefile also defines FOO and includes
Makefile.subs and hence prints out "Makefiles" for the last target "test-4".

Cheers




On 03/08/17 13:59, Benjamin Cama wrote:

...
Concerning my real use case, it is just to define target-specific
variables for some recipes which are located in subdirectories. I am not
looking for the target-specific variable feature in filenames strictly
matching the target in the current directory. I looked strange that no
variable was defined when in a subdirectory.

Anyway, it seems that a global variable would solve my problem, and
indeed it looks a bit more right this way: I can still get the stem in
my recipe (with $*), and everything works as expected. It's just that
the global namespace is “polluted” by this variable which is used only
in my particular recipe. And this is where I may be mistaken on the use
of target-specific variable: I tend to use them as a kind of “private
namespace” feature, when I define something that will be used only in
one particular recipe. But then there is no difference with global
variable in this case, apart from namespace pollution. I now understand
that this feature is better suited to overriding a global variable in
some specific recipe: is this right?

To further enhance my understanding of the “right” usage of Makefiles,
would you advise me to explicitly specify the subdirectory in the target
definition? Because I feel that counting on pattern rules to match in
subdirectory is kind of hackish, now. Like:

 $(SUBDIR)/target-%:
   …

And in my particular case, I actually know the $(SUBDIR), but would it
be right then to call make with:

 make SUBDIR=foo foo/target-1234

? It looks redundant to me.

Thanks,



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: TP-LINK Archer T2UH HELP

[Sven C. Dack]

Hello Jason,

only judging by the output you've send has the compiler been called with 
-Werror and therefore is treating warnings as errors. You may want to 
find the flag -Werror in the Makefile and remove it. See if it compiles 
with only the warnings.



On 17/08/17 00:53, th...@vmware.com wrote:

jason reynolds writes:
  >Can you help me? I am trying to setup my wifi adapter and I keep
  >getting an error.

  This mailing list is for discussions about Gnu Make.
  You are asking a specific question about how to build firmware for
  your router.  You will need to find a forum where your router and its
  firmware are discussed to obtain an answer.
Otherwise, what he says... this is a mailing list on problems with make 
itself and when its not behaving as it should be. In your case does it 
appear to be working just fine and your problem is with the source code 
you're trying to compile. Any error found in the source code by the 
compiler will cause the make process to end as it is intended to do. 
Normally do warnings by the compiler not cause an end of the make 
process, but because these warnings can hint at a possible bug in the 
code do some developers treat warnings like errors and force make to 
stop. Hence my suggestion to turn off the flag and to treat warnings 
just as warnings. If this doesn't help you then you'll have to find the 
people responsible for the source code.


Regards,
Sven


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Updated patch (was: Re: improving the -l option)

[Sven C. Dack]

Hello,

here is an updated version of the patch improving the performance of the 
-l option. It's a diff against the current git master.


The update includes O_CLOEXEC for opening the file descriptor so it 
won't "bleed" an open file descriptor to child processes. The configure 
script checks for a useable /proc/loadavg file, but the result can be 
overwritten manually with --enable-proc-loadavg={yes,no}.


Cheers,

Sven




On 09/10/16 19:58, Sven C. Dack wrote:

Hello,

I have created a patch to improve make's behaviour with regards to 
parallel execution and system load.


On Linux do the load averages refresh only every 5 seconds. Further do 
load averages only ever represent past but not present load. As a 
consequence does this lead to idle times during parallel builds where 
make is waiting for the load averages to come down. It can also spawn 
excessive amounts of processes (although it tries to balance it with 
its own heuristic).


A better way to achieve a stable load is to use the active number of 
running processes & threads over of the load averages. This number can 
be found in /proc/loadavg on Linux and is updated in real-time. An 
example:


$ cat /proc/loadavg
0.00 2.72 6.89 1/404 1890

Here it shows 1 (out of 404) running processes. The patch allows make 
to use the 4th value from /proc/loadavg.


The overall improvement for a parallel build on an 8-core CPU:

-j -l 24 (old): 269.993128122 seconds time elapsed
-j -l 24 (new): 231.300406406 seconds time elapsed

In comparison:

-j 24: 230.482726977 seconds time elapsed

The behaviour of the -l  option with the patch is now much 
closer to that of the -j  option and shows almost identical 
build times for an empty system.


The test suite passes successfully with the patch.

You'll find two patch files against make-4.2.1 in the attachment. One 
for configure.ac (includes a test for /proc/loadavg usability) and the 
other for job.c (modifies the function load_too_high()).


Regards,
Sven



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make


diff --git a/configure.ac b/configure.ac
index 1b03135..38930b7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -132,6 +132,32 @@ AS_IF([test "$ac_cv_func_gettimeofday" = yes],
 [Define to 1 if you have a standard gettimeofday function])
 ])
 
+dnl The --enable-proc-loadavg option will automatically test for the
+dnl presence of a working /proc/loadavg file, but it can also be set
+dnl manually in order to override the test result.
+AC_CACHE_CHECK(for usable /proc/loadavg, ac_cv_proc_loadavg_test,
+   [AC_RUN_IFELSE([AC_LANG_SOURCE([[
+#include 
+#include 
+#include 
+int main() {
+  int fd, i, n; char str[64];
+  fd = open("/proc/loadavg", O_RDONLY | O_CLOEXEC);
+  if (fd < 0 || read(fd, str, 64) < 0 || lseek(fd, 0, SEEK_SET) < 0) return 1;
+  for (str[63] = i = n = 0; i < 3 && n < 64; ++n) if (str[n] == ' ') ++i;
+  return i < 3 || strtoul(&str[n], NULL, 10) < 1 || close(fd) < 0; } ]])],
+  ac_cv_proc_loadavg_test=yes,
+  ac_cv_proc_loadavg_test=no)])
+
+AC_MSG_CHECKING(wether to use /proc/loadavg)
+AC_ARG_ENABLE(proc-loadavg,
+AS_HELP_STRING([--enable-proc-loadavg={yes|no}],[Allow to limit the processes started by make to not exceed the total number of active processes on a system based on /proc/loadavg [default=autodetect]]),
+ac_cv_proc_loadavg=$enableval,
+ac_cv_proc_loadavg=$ac_cv_proc_loadavg_test)
+AS_IF(test x$ac_cv_proc_loadavg = xyes,
+   AC_DEFINE(USE_PROC_LOADAVG, 1, [Defined when load limiting shall be based on /proc/loadavg]))
+AC_MSG_RESULT($ac_cv_proc_loadavg)
+
 AC_CHECK_FUNCS([strdup strndup umask mkstemp mktemp fdopen \
 dup dup2 getcwd realpath sigsetmask sigaction \
 getgroups seteuid setegid setlinebuf setreuid setregid \
diff --git a/job.c b/job.c
index 5f24597..b1ff68e 100644
--- a/job.c
+++ b/job.c
@@ -1963,6 +1963,59 @@ load_too_high (void)
 {
 #if defined(__MSDOS__) || defined(VMS) || defined(_AMIGA) || defined(__riscos__)
   return 1;
+#elif defined(USE_PROC_LOADAVG)
+  /* Read from /proc/loadavg and use the number of active running
+ tasks & threads to determine if we can start another process. */
+#define PROC_LOADAVG_SIZE 64
+  static int fd = -2;
+  char proc_loadavg[PROC_LOADAVG_SIZE];
+  int i, p;
+  unsigned long load;
+
+  if (max_load_average < 0.0 || fd == -1)
+return 0;
+
+  if (fd == -2)
+{
+  fd = open ("/proc/loadavg", O_RDONLY | O_CLOEXEC);
+  if (fd < 0)
+	{
+	  perror_with_name (_("cannot open /proc/loadavg: "), "open");
+	  fd = -1;
+	  return 0;
+	}
+}
+  p = read (fd, proc_loadavg, PROC_LOADAVG_SIZE);
+  if (p < 0)
+{
+  perror_wit

Re: [bug #52076] wildcard/glob should be sorted

[Sven C. Dack]

One thought I have here is that the sorting algorithm itself may not 
have a significant impact, but hard disk latencies may well make a 
difference.


What one sees as an (alphabetically) unordered list may well be an 
ordering significant to a file system's performance, meaning, it's not a 
completely random ordering and sorting it alphabetically can negatively 
impact the file system's efficiency. The ordering in which a file system 
returns file lists will likely be how these are physically ordered on 
the drive, thus processing them in the same exact order will help in 
reducing access latencies.


However, running builds in parallel and using SSDs over HDDs has reduced 
the latency problem significantly in recent years. This will make 
(alphabetically) unordered lists more useful to older systems, but less 
useful to newer ones.


Seeing how this isn't useful to everyone, how about making it into a 
--disable/--enable option for configure?



On 21/09/17 04:26, Bernhard M. wrote:

Follow-up Comment #2, bug #52076 (project make):

I tried to measure the performance difference using the attached 'makebench'
files, running in a tmpfs to not have physical disk IO influence results
but found that the difference is so close to zero
that it was lost in noise.


Here is what i got from running it twice each with unsorted and sorted glob:

==unsorted make:

real0m8,108s
user0m7,652s
sys 0m0,402s
make: Nothing to be done for 'run'.

real0m7,706s
user0m7,496s
sys 0m0,157s


real0m7,998s
user0m7,605s
sys 0m0,391s
make: Nothing to be done for 'run'.

real0m7,634s
user0m7,489s
sys 0m0,136s


== sorted make:

real0m8,030s
user0m7,662s
sys 0m0,372s
make: Nothing to be done for 'run'.

real0m7,583s
user0m7,417s
sys 0m0,168s


real0m8,026s
user0m7,609s
sys 0m0,423s
make: Nothing to be done for 'run'.

real0m7,646s
user0m7,509s
sys 0m0,137s


Do you have a benchmark that better shows the difference?

(file #41853)
 ___

Additional Item Attachment:

File name: makebench.tar.gz   Size:0 KB


 ___

Reply to this item at:

   

___
   Message sent via/by Savannah
   http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make




.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: A little help with Makefile.

[Sven C. Dack]

Hello Sambhav,

you've picked the wrong mailing list. The bug-make@gnu.org is a mailing 
list only for discussing bugs and issues with make program itself.


For help regarding the use of make and how to write Makefiles please do 
use help-m...@gnu.org. You can subscribe to this mailing list here:


    https://lists.gnu.org/mailman/listinfo/help-make

And do please make use of the manual. People took a lot of time in 
writing it and to provide you with as much help as possible and to give 
you examples on how to use make. Manuals are the most wonderful 
documents only waiting for you to be read. See here:


    http://www.gnu.org/software/make/manual/html_node/index.html


Cheers


On 09/10/17 15:01, Sambhav dutt wrote:

Greetings,

I am completely new to Makefile and I must run linux/unix commands and 
not C,CPP at all. I am confused how to start writing a Makefile, like 
a shell script. Can you tell me how I can write the below shell file 
in Makefile format



#!/bin/bash

mkdir -p /home/sambhav/proc
cd /home/sambhav/proc
git clone https://github.com/hyperledger/fabric.git
cd /home/sambhav/proc/fabric
make docker
cd/home/sambhav/proc
git clone https://gerrit.hyperledger.org/r/fabric-ca
cd/home/sambhav/proc/fabric-ca
make docker
cd/home/sambhav/proc/fabric-ca
make docker
bash 
/home/sambhav/fabric-test/fabric-test/feature/scripts/install_behave.sh


===

Thank You,


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Converting make rules into other file formats

[Sven C. Dack]

Hello Markus,

you've picked the wrong mailing list. The bug-make@gnu.org is a mailing 
list only for discussing bugs and issues with the make program itself. 
Please use help-m...@gnu.org for your question.


You can subscribe to help mailing list here:

https://lists.gnu.org/mailman/listinfo/help-make


Regarding your question itself, I'd say you're looking at two completely 
different worlds. A Makefile is basically a human-readable text file 
with instructions on how to create executables with a compiler for 
example. It has pretty much nothing to do with creating graphics. You 
will likely have to be more specific when you ask your question on 
bug-make@gnu.org.


Cheers



On 14/10/17 13:40, SF Markus Elfring wrote:

Hello,

How does the current software situation look like for the conversion of data
from make scripts into file formats with which other users are more familiar?

Examples:

* Outlines

* Vector graphics


Would you like to point any special development tools out?

Regards,
Markus

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Converting make rules into other file formats

[Sven C. Dack]

I apologise for the mix up. I've pasted the wrong e-mail address into my 
response. I meant to say that you will have to be more specific when you 
ask on help-m...@gnu.org.


If you could please continue the conversation there, thank you.


On 14/10/17 17:06, Sven C. Dack wrote:
... You will likely have to be more specific when you ask your 
question on bug-make@gnu.org.




___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Regression caused by Commit: 5bd7ad2b225b ("Preserve the real value of -jN in MAKEFLAGS using jobserver.")

[Sven C. Dack]

On 30/10/17 07:33, Tim Murphy wrote:
For my money -j is always a top-level setting.   To have it per 
makefile and then let makefiles depend on that behaviour would open 
the door to lots of scary bugs.


Regards,

Tim


Agreed. Using -j in a Makefile is not portable, because it can cause a 
system to run out of memory or just start an unwanted and excessive 
amount of jobs.


However, if someone wanted to control the number of jobs from within a 
Makefile then this shouldn't be discouraged either. Nor does make give 
any kind of guarantees on how many jobs are started by other processes, 
because this isn't something, which can be controlled easily from within 
make, and I don't believe many expect such a form of control by make.


We do have flags such as -n, -k and -S, where users expect a strict 
behaviour for -n, but not so for -k and -S (continue on errors, stop on 
errors). The later two can both be set from within Makefiles, with -k 
being potentially dangerous if set from within.


While I dislike the use of -j in Makefiles is it not as bad as if having 
a (hypothetical) option to disable -n for example. As such do I see it 
being in the same boat with -k. Any problems caused by using -j from 
within Makefiles are the responsibility of the person who put it there. 
It shouldn't be made a responsibility of the devs of make unless we have 
very good reasons to do so.


Cheers,
Sven


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Regression caused by Commit: 5bd7ad2b225b ("Preserve the real value of -jN in MAKEFLAGS using jobserver.")

[Sven C. Dack]

On 30/10/17 21:22, Paul Smith wrote:

If we see -j in a makefile setting of MAKEFLAGS, we could:

1. Always silently ignore it (today's behavior).
2. Always print a message then ignore it.
3. Treat it the same way as a recipe with $(MAKE) -j: that is, start a
   new jobserver group for this sub-make (and print a message if we're
   already part of a different jobserver group).

Personally I lean towards #3, even if it does allow people to do
bizarre things.


I like 3, too. It gives more options to Makefile authors. We will only 
see more parallelism in the future, with more cores per CPU and people 
trying to utilise the hardware anyway they can. Who knows how helpful it 
might become.


Also, the -l option (load limiting) when set should get passed along to 
make it more meaningful - unless it, too, receives a new value for 
whatever reason.



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Linux /proc/loadavg

[Sven C. Dack]

Hello,

the ChangeLog says the following:

2019-12-17  Paul Smith 

...

    * src/job.c (load_too_high): Disable Linux /proc/loadavg for now
    This new feature has a problem: if you provide a load which is
    larger than the number of cores then it will always run every job.
    Before we can enable it we need to at the least learn how to clamp
    this value to the number of cores.  To experiment with it, set
    PROC_FD_INIT to -2 in job.c to re-enable the feature.


I have recently been looking into this again, because of a new feature I
am currently testing, and so far can only repeat what I have already
said in the past. The /proc/loadavg file under Linux with a vanilla
kernel is behaving as expected, meaning, the number of active processes
can very well exceed the number of cores.

To quote from the Linux source code documentation under
Documentation/filesystems/proc.rst:

 loadavg  Load average of last 1, 5 & 15 minutes;
    number of processes currently runnable (running or on
ready queue);
    total number of processes in system;
    last pid created.
    All fields are separated by one space except "number of
    processes currently runnable" and "total number of
processes
    in system", which are separated by a slash ('/'). Example:
    0.61 0.61 0.55 3/828 22084

Here it states that it is the number of running or ready processes. Any
implementation of /proc/loadavg which counts only one running process
per core is effectively counting the number of active cores, but not the
number of running or runnable processes, and should therefore be
considered buggy or deviant.

Thus, when there is a deviant implementation of /proc/loadavg out there,
by a modified Linux kernel or perhaps another OS, then we should gate
this off either through autoconf or a simple #ifdef linux #endif.

Sven

Re: Linux /proc/loadavg (graphs)

[Sven C. Dack]

Here are some graphs. These show the difference when using Linux
/proc/loadavg. Red on the left shows the default behaviour of make-4.3
when using the -l option, green on the right shows the modified
behaviour. The upper graphs show the number of processes on the system,
the lower graphs show the memory use.

https://i.imgur.com/9oshAfc.png


On 15/01/2022 10:29, Sven C. Dack wrote:

Hello,

the ChangeLog says the following:

2019-12-17  Paul Smith 

...

    * src/job.c (load_too_high): Disable Linux /proc/loadavg for now
    This new feature has a problem: if you provide a load which is
    larger than the number of cores then it will always run every job.
    Before we can enable it we need to at the least learn how to clamp
    this value to the number of cores.  To experiment with it, set
    PROC_FD_INIT to -2 in job.c to re-enable the feature.


I have recently been looking into this again, because of a new feature I
am currently testing, and so far can only repeat what I have already
said in the past. The /proc/loadavg file under Linux with a vanilla
kernel is behaving as expected, meaning, the number of active processes
can very well exceed the number of cores.

To quote from the Linux source code documentation under
Documentation/filesystems/proc.rst:

 loadavg  Load average of last 1, 5 & 15 minutes;
    number of processes currently runnable (running or on
ready queue);
    total number of processes in system;
    last pid created.
    All fields are separated by one space except "number of
    processes currently runnable" and "total number of
processes
    in system", which are separated by a slash ('/').
Example:
    0.61 0.61 0.55 3/828 22084

Here it states that it is the number of running or ready processes. Any
implementation of /proc/loadavg which counts only one running process
per core is effectively counting the number of active cores, but not the
number of running or runnable processes, and should therefore be
considered buggy or deviant.

Thus, when there is a deviant implementation of /proc/loadavg out there,
by a modified Linux kernel or perhaps another OS, then we should gate
this off either through autoconf or a simple #ifdef linux #endif.

Sven

[PATCH] new option -u, --memory-used

[Sven C. Dack]

For Your Consideration
==

Hello,

this is an implementation of a new option for Make. It controls the
number of parallel jobs in regards to the amount of memory in use on a
host. It allows one to use Make's parallel job execution on hosts with
multiple CPU cores, but where memory is limited. It works similar to
the -l, --load-average option and has to be used together with the -j,
--jobs option.

For example, assuming a host with 2GB of memory, the following options
are equivalent:

    -u25%, -u0.25, -u512m, or -u0.5g

In this example will Make stop creating new jobs when a host's memory
use reaches 512MB (-j1), and will resume when it drops back below the
threshold (-jN). A memory threshold can be specified as either an
absolute value or as a percentage of the host's total memory.

When the option is used in combination with -l, --load-average then
Make stops creating new jobs when either the load threshold has been
reached, or when the memory threshold has been reached. In addition
will Make use the load and memory thresholds as a hint for a job's
average memory use and scales the number of jobs (from -j1 to -jN).
This reduces load and memory spikes near the thresholds and stabilizes
the memory usage of a Make run.

A value of 0 turns it off (-u0) and sets it to unlimited, which is
also the default.

The attached patch is against release 4.3 of Make. It implements the
-u option for Linux using /proc/meminfo, and it enables the use of
/proc/loadavg for the -l option.


How well does it work?
==

To show its effectiveness in practise did I create graphs for system
loads (upper graphs) and memory usages (lower graphs) of different
Make runs. In these graphs is an average shown (thicker line) together
with the raw values (thinner lines, spikes). Because the memory use
can vary between runs did I animate the graphs by using 5 Make runs.
The "std-" labels in the legends mean that the standard Make 4.3
release was used. The "new-" labels mean the patched version was used.

The following is a Linux kernel build on a 16-core CPU with 32GB of
memory. The first graphs (black) show standard runs with only the -j
option. The second graphs (red) show standard runs with -j and -l. The
third graphs (green) show runs with -j and -l, using /proc/loadavg
under Linux. The forth graphs (blue) show runs with -j, -l and -u
combined, using /proc/loadavg and /proc/meminfo. The additional graphs
show variations of the arguments to the -u option:

https://zippyimage.com/images/2022/02/12/183964b428e4a0f0530c647c923924b9.gif



The same kernel build on a 4-core SoC with 512MB:

https://zippyimage.com/images/2022/02/12/aa05eb906c4bb1f6eb969b5bc29b7731.gif



To test the limits did I use a Makefile from image processing. It
consists of 3 rules and processes 30 files, leaving little room for
Make to control its jobs. The jobs themselves use multiple threads,
which inflate the system load, and use a fair amount of memory:

https://zippyimage.com/images/2022/02/12/189b22aa75adc0a6b7a34b6b9cceabee.gif



A more practical test is a build of LLVM/Clang-13 with itself, using
link time optimization (LTO). The Makefiles are CMake-generated. Here
a link job with LTO can use up to 24GB of memory. The host itself has
got 32GB of RAM plus 16GB of ZRAM for swap, making it near impossible
to build with a fixed number of parallel jobs. A "make -j8" will
consistently run out of memory and out of swap space (red, failed). A
"make -j8 -u30%" will build it 5 times without failing. Other values
for -u work, too, however only builds, which succeeded at least 5
times and not failed once, are shown in the graphs. The combination of
-l with -u together works better and allows to build with "-j16 -l16
-u90%" in under an hour:

https://zippyimage.com/images/2022/02/12/9837a5551cc9350c17806c7b57efb991.gif



The last test is a LLVM/Clang-13 LTO build with "-j16 -l16 -u90%" and
CCACHE enabled. It speeds up the build time and brings the LTO link
jobs closer together, causing them to further overlap. It builds 5
times without failing:

https://zippyimage.com/images/2022/02/12/a55d2c3352a134edcb6af3feed8c2ff6.gif



The patch is provided "as is". I hope you enjoy it and find it useful.

Regards,
Sven
diff -r -u make-4.3/src/job.c make-4.3.1/src/job.c
--- make-4.3/src/job.c  2020-01-19 20:32:59.0 +
+++ make-4.3.1/src/job.c2022-02-11 14:09:08.074210132 +
@@ -221,6 +221,7 @@
 static void free_child (struct child *);
 static void start_job_command (struct child *child);
 static int load_too_high (void);
+static int memory_too_high (void);
 static int job_next_command (struct child *);
 static int start_waiting_job (struct child *);
 
@@ -1616,7 +1617,7 @@
   /* If we are running at least one job already and the load average
  is too high, make this one wait.  */
   if (!c->remote
-  

Re: Goodbye to GNU make's "build.sh" ... ?

[Sven C. Dack]

Hello,

developers causing a "Catch 22" paradox is not new. You want to avoid
such software, or at least work around it with a big margin. Sharply
limiting the use of gnulib here may not provide such a margin and cause
repeated trouble in the future. Your condition of "to those with
trivial-enough configurations" can fail with the next update to gnulib.

How about a third option:

3. Allow to build a minimal Make executable, which provides basic and
traditional Make functionality and does not rely on gnulib, and then use
it as a bootstrap.

Then make the gnulib maintainers aware of the "Catch 22" and tell them
not rely on newer GNU Make versions. Offering a minimal Make executable
next to the full GNU Make may also have uses for other projects.

-Sven


On 25/06/2022 22:47, Paul Smith wrote:

I'm trying to decide what the future is for GNU make's "build.sh"
bootstrapping script. As you may recall, this script is provided to
allow GNU make to build on systems which don't already have an instance
of make installed. Its goal is to build the first make binary, without
of course all the fancy parts of avoiding rebuilds, generating
dependency files, etc.

Unfortunately, this really cannot work if GNU make is going to rely on
gnulib, because a MANY gnulib modules require not only autoconf to
work, but also automake. That is, it's not enough to just run configure
and then you get a set of source files and header files that you can
compile by hand by just invoking the compiler. You must ALSO run make,
because the modules provide makefile recipes that invoke sed, etc. to
convert files into their final form.

I had a discussion about this with the Gnulib maintainers a while ago:

https://lists.gnu.org/archive/html/bug-gnulib/2019-09/msg00041.html

However the gnulib maintainers were disinclined to modify the practices
of the gnulib modules.

This leaves me with two options:

1. Stop using gnulib, or at least sharply limit the modules we will
   include to those with trivial-enough configurations.
2. Abandon the build.sh script and require an existing make program
   in order to build a new version of GNU make.

#1 is what I followed for GNU make 4.3, which has a limited subset of
carefully-chosen modules. However this becomes harder over time. For
example any module that needs unistd.h requires a very complex automake
rule.

If #2 is chosen, then a bootstrap process would involve first obtaining
an older version of make, such as GNU make 4.3 or lower, and building
that with its build.sh, then using the resulting make to build the
newer version.

I'm interested in thoughts about these options.