Re: AIX and Interix also do early PID recycling.
On 07/25/2012 03:05 AM, Chet Ramey wrote: > Bash assumes that there's a PID space at least as > large as CHILD_MAX, and that the kernel will use all of it before reusing > any PID in the space. Posix says that shells must remember up to CHILD_MAX > statuses of terminated asynchronous children (the description of `wait'), > so implicitly the kernel is not allowed to reuse process IDs until it has > exhausted CHILD_MAX PIDs. What about grand-childs? They do count for the kernel, but not for the toplevel shell... > The description of fork() doesn't mention this, > however. The Posix fork() requirement that the PID returned can't > correspond to an existing process or process group is not sufficient to > satisfy the requirement on `wait'. OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by fork(). However, I'm unable to find that in the POSIX spec. > Bash holds on to the status of all terminated processes, not just > background ones, and only checks for the presence of a newly-forked PID > in that list if the list size exceeds CHILD_MAX. One of the results of > defining RECYCLES_PIDS is that the check is performed on every created > process. What if the shell does not do waitpid(-1), but waitpid(known-child-PID). That would mean to waitpid(synchronous-child-PID) immediately, and waitpid(asynchronous-child-PID) upon some "wait $!" shell command, rendering to waitpid(-1) when there's no PID passed to "wait". > I'd be interested in knowing the value of CHILD_MAX (or even `ulimit -c') > on the system where you're seeing this problem. The AIX 6.1 I've debugged on has: #define CHILD_MAX 128 #define _POSIX_CHILD_MAX 25 sysconf(_SC_CHILD_MAX) = 1024 $ ulimit -H -c -u core file size (blocks, -c) unlimited max user processes (-u) unlimited $ ulimit -S -c -u core file size (blocks, -c) 1048575 max user processes (-u) unlimited The Interix 6.1 we do have similar-looking stability problems has: CHILD_MAX not defined #define _POSIX_CHILD_MAX 6 sysconf(_SC_CHILD_MAX) = 512 $ ulimit -H -c -u core file size (blocks, -c) unlimited max user processes (-u) 512 $ ulimit -S -c -u core file size (blocks, -c) unlimited max user processes (-u) 512 > The case where last_made_pid is equal to last_pid is a problem only when > the PID space is extremely small -- on the order of, say, 4 -- as long as > the kernel behaves as described above. I'm going to run this build job with 'truss -t kfork' again, to eventually find some too small count of different PIDs before PID-recycling by the kernel... Anyway - defining RECYCLES_PIDS for that AIX 6.1 has reduced the error rate for this one build job from ~37 to 0 when run 50 times. /haubi/
Re: AIX and Interix also do early PID recycling.
On Wed, Jul 25, 2012 at 09:59:28AM +0200, Michael Haubenwallner wrote: > OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by > fork(). > However, I'm unable to find that in the POSIX spec. A process that hasn't been waited for should become a zombie, which should be sufficient to prevent its PID being reused. Are you saying that AIX and Interix don't have zombies?
Re: AIX and Interix also do early PID recycling.
On 7/25/2012 8:14 AM, Greg Wooledge wrote: On Wed, Jul 25, 2012 at 09:59:28AM +0200, Michael Haubenwallner wrote: OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by fork(). However, I'm unable to find that in the POSIX spec. A process that hasn't been waited for should become a zombie, which should be sufficient to prevent its PID being reused. Are you saying that AIX and Interix don't have zombies? One small nit: A process that hasn't been waited for should become a zombie *if* the parent dies *or* the parent has set SIGCHLD to SIG_IGN. If the parent persists and the children are not harvested then the children are in a defunct state. I don't know what Internix is, but I once discovered an AIX system that had over 10,000 defunct processes. The people would detect a slowdown every few days and just reboot it. Turns out that a programmer had written a server that did not manage its slaves, so they all just sat there waiting to be waited for. I went in and added a few lines of code to do the wait calls. It never needed a reboot since. -- Time flies like the wind. Fruit flies like a banana. Stranger things have .0. happened but none stranger than this. Does your driver's license say Organ ..0 Donor?Black holes are where God divided by zero. Listen to me! We are all- 000 individuals! What if this weren't a hypothetical question? steveo at syslang.net
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 09:59 AM, Michael Haubenwallner wrote: > On 07/25/2012 03:05 AM, Chet Ramey wrote: >> Bash holds on to the status of all terminated processes, not just >> background ones, and only checks for the presence of a newly-forked PID >> in that list if the list size exceeds CHILD_MAX. > The AIX 6.1 I've debugged on has: > #define CHILD_MAX 128 > I'm going to run this build job with 'truss -t kfork' again, to eventually > find > some too small count of different PIDs before PID-recycling by the kernel... Tracing shows: The minimum fork count (including grand-childs to any depth) before PID recycling starts looks like 255 (once), but usually 256 and more. However, one process does see a PID recycled after *at least* 128 forks, that is exactly the value of CHILD_MAX. First thought is of some off-by-one bug, but reducing js.c_childmax in jobs.c (2 times) by one doesn't help. Investigating further... any hints what to look out for? /haubi/
Re: AIX and Interix also do early PID recycling.
On 7/25/2012 9:20 AM, Michael Haubenwallner wrote:
On 07/25/2012 09:59 AM, Michael Haubenwallner wrote:
On 07/25/2012 03:05 AM, Chet Ramey wrote:
Bash holds on to the status of all terminated processes, not just
background ones, and only checks for the presence of a newly-forked PID
in that list if the list size exceeds CHILD_MAX.
The AIX 6.1 I've debugged on has:
#define CHILD_MAX 128
I'm going to run this build job with 'truss -t kfork' again, to eventually find
some too small count of different PIDs before PID-recycling by the kernel...
Tracing shows:
The minimum fork count (including grand-childs to any depth) before PID
recycling starts
looks like 255 (once), but usually 256 and more.
However, one process does see a PID recycled after *at least* 128 forks,
that is exactly the value of CHILD_MAX.
First thought is of some off-by-one bug, but reducing js.c_childmax in jobs.c
(2 times)
by one doesn't help.
Investigating further... any hints what to look out for?
/haubi/
One thing that just springs to mind: Under AIX, I believe (but don't take this
as gospel) that PIDs are not strictly ascending. The low byte may increment,
but it might be another byte in the pid_t that gets the increment, and the
byte that increments may alternate. I don't know if anyone else does this, but
I seem to remember that AIX does (or did).
You might write a little C program that does a loop of 1000 fork and wait
calls. Just print out the pids that you create. Something like this...
#include
#include
#include
main()
{
pid_t pid;
int status;
int ii;
for (ii = 0; ii < 1000; ii++)
{
if ((pid = fork()) != 0)
{
printf("New child:%d\n", pid);
waitpid(pid, &status, 0);
}
else
exit(0);
}
}
--
Time flies like the wind. Fruit flies like a banana. Stranger things have .0.
happened but none stranger than this. Does your driver's license say Organ ..0
Donor?Black holes are where God divided by zero. Listen to me! We are all- 000
individuals! What if this weren't a hypothetical question?
steveo at syslang.net
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 02:14 PM, Greg Wooledge wrote: > On Wed, Jul 25, 2012 at 09:59:28AM +0200, Michael Haubenwallner wrote: >> OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by >> fork(). >> However, I'm unable to find that in the POSIX spec. > > A process that hasn't been waited for should become a zombie, which > should be sufficient to prevent its PID being reused. Are you saying > that AIX and Interix don't have zombies? Nope. My thought was that bash eventually could postpone waiting for a specific child PID until required by the driving shell script. That is: immediately for synchronous childs to set $?, and on "wait" for asynchronous childs. The idea was to render storing CHILD_MAX returnvalues obsolete. However, I'm investigating why respecting CHILD_MAX by bash doesn't work when the kernel starts reusing PIDs after CHILD_MAX different ones. /haubi/
Re: AIX and Interix also do early PID recycling.
On 7/25/12 3:59 AM, Michael Haubenwallner wrote: > > On 07/25/2012 03:05 AM, Chet Ramey wrote: >> Bash assumes that there's a PID space at least as >> large as CHILD_MAX, and that the kernel will use all of it before reusing >> any PID in the space. Posix says that shells must remember up to CHILD_MAX >> statuses of terminated asynchronous children (the description of `wait'), >> so implicitly the kernel is not allowed to reuse process IDs until it has >> exhausted CHILD_MAX PIDs. > > What about grand-childs? > They do count for the kernel, but not for the toplevel shell... Yes, that's another problem. >> The description of fork() doesn't mention this, >> however. The Posix fork() requirement that the PID returned can't >> correspond to an existing process or process group is not sufficient to >> satisfy the requirement on `wait'. > > OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by > fork(). > However, I'm unable to find that in the POSIX spec. If the process hasn't been reaped by its parent, it's still technically active, and its PID is not supposed to be eligible for reuse. > >> Bash holds on to the status of all terminated processes, not just >> background ones, and only checks for the presence of a newly-forked PID >> in that list if the list size exceeds CHILD_MAX. One of the results of >> defining RECYCLES_PIDS is that the check is performed on every created >> process. > > What if the shell does not do waitpid(-1), but waitpid(known-child-PID). > That would mean to waitpid(synchronous-child-PID) immediately, and > waitpid(asynchronous-child-PID) upon some "wait $!" shell command, rendering > to waitpid(-1) when there's no PID passed to "wait". That's not how the shell is architected. In your scenario, the shell would ignore process status changes until and unless the script asked for them. You'd never reap processes that were begun to run command substitutions, for example, and async processes would linger until the script happened to ask for some status (which few do). > >> I'd be interested in knowing the value of CHILD_MAX (or even `ulimit -c') >> on the system where you're seeing this problem. > > The AIX 6.1 I've debugged on has: > #define CHILD_MAX 128 > #define _POSIX_CHILD_MAX 25 > sysconf(_SC_CHILD_MAX) = 1024 > > $ ulimit -H -c -u > core file size (blocks, -c) unlimited > max user processes (-u) unlimited > > $ ulimit -S -c -u > core file size (blocks, -c) 1048575 > max user processes (-u) unlimited Bash prefers sysconf(_SC_CHILD_MAX) and will use it over the other defines (lib/sh/oslib.c:getmaxchild()). I don't know why AIX chooses to return a different value via sysconf than it defines for CHILD_MAX, especially when it seems to use the CHILD_MAX value to decide when it can recycle the PID space. >> The case where last_made_pid is equal to last_pid is a problem only when >> the PID space is extremely small -- on the order of, say, 4 -- as long as >> the kernel behaves as described above. > > I'm going to run this build job with 'truss -t kfork' again, to eventually > find > some too small count of different PIDs before PID-recycling by the kernel... > > Anyway - defining RECYCLES_PIDS for that AIX 6.1 has reduced the error rate > for > this one build job from ~37 to 0 when run 50 times. And I suspect that the single change of significance is to not check against the childmax value when deciding whether or not to look for and remove this pid from the list of saved termination status values. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: AIX and Interix also do early PID recycling.
On 7/25/12 8:14 AM, Greg Wooledge wrote: > On Wed, Jul 25, 2012 at 09:59:28AM +0200, Michael Haubenwallner wrote: >> OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by >> fork(). >> However, I'm unable to find that in the POSIX spec. > > A process that hasn't been waited for should become a zombie, which > should be sufficient to prevent its PID being reused. Are you saying > that AIX and Interix don't have zombies? No, a slightly different problem. Bash interprets the Posix text for the `wait' builtin as guaranteeing that the kernel will wait at least CHILD_MAX children before reusing a PID, whether they've terminated or not. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 03:20 PM, Michael Haubenwallner wrote: > On 07/25/2012 09:59 AM, Michael Haubenwallner wrote: >> On 07/25/2012 03:05 AM, Chet Ramey wrote: >>> Bash holds on to the status of all terminated processes, not just >>> background ones, and only checks for the presence of a newly-forked PID >>> in that list if the list size exceeds CHILD_MAX. > >> The AIX 6.1 I've debugged on has: >> #define CHILD_MAX 128 >> #define _POSIX_CHILD_MAX 25 >> sysconf(_SC_CHILD_MAX) = 1024 > Tracing shows: > > The minimum fork count (including grand-childs to any depth) before PID > recycling starts > looks like 255 (once), but usually 256 and more. > > However, one process does see a PID recycled after *at least* 128 forks, > that is exactly the value of CHILD_MAX. Got it: The value used for js.c_childmax isn't 128, but 1024. In lib/sh/oslib.c, getmaxchild() prefers sysconf(_SC_CHILD_MAX) over CHILD_MAX over MAXUPRC. But sysconf(_SC_CHILD_MAX) does return the number of "processes per real user id" (similar to ulimit -u), rather than the number of CHILD_MAX (whenever defined). For Interix, things are different though: There is no CHILD_MAX nor MAXUPRC defined, and sysconf(_SC_CHILD_MAX) does return 512, but PIDs start to be recycled at ~120 already... Any idea about the "correct" fix for getmaxchild() across platforms? /haubi/
Re: AIX and Interix also do early PID recycling.
On 7/25/12 9:20 AM, Michael Haubenwallner wrote: > > On 07/25/2012 09:59 AM, Michael Haubenwallner wrote: >> On 07/25/2012 03:05 AM, Chet Ramey wrote: >>> Bash holds on to the status of all terminated processes, not just >>> background ones, and only checks for the presence of a newly-forked PID >>> in that list if the list size exceeds CHILD_MAX. > >> The AIX 6.1 I've debugged on has: >> #define CHILD_MAX 128 > >> I'm going to run this build job with 'truss -t kfork' again, to eventually >> find >> some too small count of different PIDs before PID-recycling by the kernel... > > Tracing shows: > > The minimum fork count (including grand-childs to any depth) before PID > recycling starts > looks like 255 (once), but usually 256 and more. > > However, one process does see a PID recycled after *at least* 128 forks, > that is exactly the value of CHILD_MAX. > > First thought is of some off-by-one bug, but reducing js.c_childmax in jobs.c > (2 times) > by one doesn't help. It wouldn't. Bash prefers sysconf(_SC_CHILD_MAX) to the CHILD_MAX define, and those two are apparently different. You could replace js.c_childmax in tests against its value (three places in jobs.c) with CHILD_MAX and see whether that makes a difference. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: AIX and Interix also do early PID recycling.
On 7/25/12 9:44 AM, Steven W. Orr wrote: > One thing that just springs to mind: Under AIX, I believe (but don't take > this as gospel) that PIDs are not strictly ascending. The low byte may > increment, but it might be another byte in the pid_t that gets the > increment, and the byte that increments may alternate. I don't know if > anyone else does this, but I seem to remember that AIX does (or did). That does not (or should not) matter. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 04:52 PM, Michael Haubenwallner wrote: Got it: The value used for js.c_childmax isn't 128, but 1024. In lib/sh/oslib.c, getmaxchild() prefers sysconf(_SC_CHILD_MAX) over CHILD_MAX over MAXUPRC. But sysconf(_SC_CHILD_MAX) does return the number of "processes per real user id" (similar to ulimit -u), rather than the number of CHILD_MAX (whenever defined). For Interix, things are different though: There is no CHILD_MAX nor MAXUPRC defined, and sysconf(_SC_CHILD_MAX) does return 512, but PIDs start to be recycled at ~120 already... Any idea about the "correct" fix for getmaxchild() across platforms? /haubi/ And there is also a problem when you change the `ulimit -u' value. See https://bugzilla.redhat.com/show_bug.cgi?id=832997 RR
Re: AIX and Interix also do early PID recycling.
On 7/25/12 10:11 AM, Michael Haubenwallner wrote: > > On 07/25/2012 02:14 PM, Greg Wooledge wrote: >> On Wed, Jul 25, 2012 at 09:59:28AM +0200, Michael Haubenwallner wrote: >>> OTOH, AFAICT, as long as a PID isn't waitpid()ed for, it isn't reused by >>> fork(). >>> However, I'm unable to find that in the POSIX spec. >> >> A process that hasn't been waited for should become a zombie, which >> should be sufficient to prevent its PID being reused. Are you saying >> that AIX and Interix don't have zombies? > > Nope. My thought was that bash eventually could postpone waiting for a > specific > child PID until required by the driving shell script. That is: immediately for > synchronous childs to set $?, and on "wait" for asynchronous childs. The idea > was to render storing CHILD_MAX returnvalues obsolete. It won't; Posix makes it required. However, Posix does allow shells to detect when $! is referenced and not save that many termination statuses if it's not. The problem is as I explained in a previous message: you have processes you have to reap whether or not the script calls `wait'. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: AIX and Interix also do early PID recycling.
Chet Ramey writes: > And that's the problem. Bash assumes that there's a PID space at least as > large as CHILD_MAX, and that the kernel will use all of it before reusing > any PID in the space. Posix says that shells must remember up to CHILD_MAX > statuses of terminated asynchronous children (the description of `wait'), > so implicitly the kernel is not allowed to reuse process IDs until it has > exhausted CHILD_MAX PIDs. I cannot see how CHILD_MAX is related to pid reuse. CHILD_MAX is a per-user limit, but the pid namespace is global. If the shell forks a new process, and the pid of it matches one of the previously used pids for asynchronous jobs it can surely discard the remembered status for that job. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Re: AIX and Interix also do early PID recycling.
On 07/25/2012 04:50 PM, Chet Ramey wrote: >> The AIX 6.1 I've debugged on has: >> #define CHILD_MAX 128 >> #define _POSIX_CHILD_MAX 25 >> sysconf(_SC_CHILD_MAX) = 1024 > Bash prefers sysconf(_SC_CHILD_MAX) and will use it over the other > defines (lib/sh/oslib.c:getmaxchild()). I don't know why AIX chooses > to return a different value via sysconf than it defines for CHILD_MAX, > especially when it seems to use the CHILD_MAX value to decide when it > can recycle the PID space. Well, _SC_CHILD_MAX is documented across platforms as: (Linux) "The max number of simultaneous processes per user ID." (HP-UX) "Maximum number of simultaneous processes per user ID." (Solaris) "Max processes allowed to a UID" (AIX) "Specifies the number of simultaneous processes per real user ID." (Interix) "Maximum number of simultaneous processes per user ID." Also, one Linux machine actually shows the _SC_CHILD_MAX value equal to kernel.pid_max (32768 here), so even Linux could see this problem in theory, because PIDs really are recycled before kernel.pid_max. > And I suspect that the single change of significance is to not check > against the childmax value when deciding whether or not to look for and > remove this pid from the list of saved termination status values. Agreed - but is this still different to defining RECYCLES_PIDS then? Thank you! /haubi/
Re: AIX and Interix also do early PID recycling.
On 7/25/12 11:59 AM, Michael Haubenwallner wrote: > > On 07/25/2012 04:50 PM, Chet Ramey wrote: >>> The AIX 6.1 I've debugged on has: >>> #define CHILD_MAX 128 >>> #define _POSIX_CHILD_MAX 25 >>> sysconf(_SC_CHILD_MAX) = 1024 > >> Bash prefers sysconf(_SC_CHILD_MAX) and will use it over the other >> defines (lib/sh/oslib.c:getmaxchild()). I don't know why AIX chooses >> to return a different value via sysconf than it defines for CHILD_MAX, >> especially when it seems to use the CHILD_MAX value to decide when it >> can recycle the PID space. > > Well, _SC_CHILD_MAX is documented across platforms as: > (Linux) "The max number of simultaneous processes per user ID." > (HP-UX) "Maximum number of simultaneous processes per user ID." > (Solaris) "Max processes allowed to a UID" > (AIX) "Specifies the number of simultaneous processes per real user ID." > (Interix) "Maximum number of simultaneous processes per user ID." > > Also, one Linux machine actually shows the _SC_CHILD_MAX value equal to > kernel.pid_max (32768 here), so even Linux could see this problem in theory, > because PIDs really are recycled before kernel.pid_max. That's interesting, since Posix describes sysconf() as simply a way to retrieve values from limits.h or unistd.h that one wishes to get at run time rather than compile time. And interesting that it establishes a correspondence between CHILD_MAX and _SC_CHILD_MAX. > >> And I suspect that the single change of significance is to not check >> against the childmax value when deciding whether or not to look for and >> remove this pid from the list of saved termination status values. > > Agreed - but is this still different to defining RECYCLES_PIDS then? It is not. It is one of the things that happens when you define RECYCLES_PIDS. The question is whether or not that is the single thing that makes a difference in this case. If it is, there is merit in removing the check against js.c_childmax entirely or making it dependent on something else. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: AIX and Interix also do early PID recycling.
On 7/25/12 11:33 AM, Andreas Schwab wrote: > I cannot see how CHILD_MAX is related to pid reuse. CHILD_MAX is a > per-user limit, but the pid namespace is global. If the shell forks a > new process, and the pid of it matches one of the previously used pids > for asynchronous jobs it can surely discard the remembered status for > that job. Thanks, that's a good restatement of the problem. Your proposed solution is one of the possibles. The question is whether or not it's necessary (apparently on some systems) and sufficient (probably). Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: AIX and Interix also do early PID recycling.
On 7/25/12 10:57 AM, Roman Rakus wrote: > And there is also a problem when you change the `ulimit -u' value. > See https://bugzilla.redhat.com/show_bug.cgi?id=832997 It's not clear to me that it's desirable to attempt to remember a potentially unlimited number of process exit statuses. You have to bound it somewhere. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
