Re: Parallelism a la make -j / GNU parallel
On 5/3/12 2:49 PM, Colin McEwan wrote:
> What I would really *like* would be an extension to the shell which
> implements the same sort of parallelism-limiting / 'process pooling' found
> in make or 'parallel' via an operator in the shell language, similar to '&'
> which has semantics of *possibly* continuing asynchronously (like '&') if
> system resources allow, or waiting for the process to complete (';').
I think the combination of asynchronous jobs and `wait' provides most of
what you need. The already-posted alternatives look like a good start to a
general solution.
If those aren't general enough, how would you specify the behavior of a
shell primitive -- operator or builtin -- that does what you want?
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Parallelism a la make -j / GNU parallel
On Friday 04 May 2012 08:55:42 Chet Ramey wrote:
> On 5/3/12 2:49 PM, Colin McEwan wrote:
> > What I would really *like* would be an extension to the shell which
> > implements the same sort of parallelism-limiting / 'process pooling'
> > found in make or 'parallel' via an operator in the shell language,
> > similar to '&' which has semantics of *possibly* continuing
> > asynchronously (like '&') if system resources allow, or waiting for the
> > process to complete (';').
>
> I think the combination of asynchronous jobs and `wait' provides most of
> what you need. The already-posted alternatives look like a good start to a
> general solution.
>
> If those aren't general enough, how would you specify the behavior of a
> shell primitive -- operator or builtin -- that does what you want?
i wish there was a way to use `wait` that didn't block until all the pids
returned. maybe a dedicated option, or a shopt to enable this, or a new
command.
for example, if i launched 10 jobs in the background, i usually want to wait
for the first one to exit so i can queue up another one, not wait for all of
them.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: Parallelism a la make -j / GNU parallel
On Fri, May 04, 2012 at 12:41:03PM -0400, Mike Frysinger wrote: > i wish there was a way to use `wait` that didn't block until all the pids > returned. maybe a dedicated option, or a shopt to enable this, or a new > command. wait takes arguments. > for example, if i launched 10 jobs in the background, i usually want to wait > for the first one to exit so i can queue up another one, not wait for all of > them. Do you mean "for *any* one of them", or literally "for the first one"? The latter, you can do right now -- just pass the PID of the first one. The former would require that you set up a SIGCHLD trap and do some work. By the way, "help wait" is misleading; it says you can only pass a single job ID argument. The man page indicates that you can pass multiple arguments.
Re: Parallelism a la make -j / GNU parallel
On Friday 04 May 2012 12:44:32 Greg Wooledge wrote: > On Fri, May 04, 2012 at 12:41:03PM -0400, Mike Frysinger wrote: > > i wish there was a way to use `wait` that didn't block until all the pids > > returned. maybe a dedicated option, or a shopt to enable this, or a new > > command. > > wait takes arguments. yes, and it'll wait for all the ones i specified before returning > > for example, if i launched 10 jobs in the background, i usually want to > > wait for the first one to exit so i can queue up another one, not wait > > for all of them. > > Do you mean "for *any* one of them", or literally "for the first one"? any. maybe `wait -1` will translate into waitpid(-1, ...). -mike signature.asc Description: This is a digitally signed message part.
Re: Parallelism a la make -j / GNU parallel
Mike Frysinger writes: > i wish there was a way to use `wait` that didn't block until all the pids > returned. maybe a dedicated option, or a shopt to enable this, or a new > command. > > for example, if i launched 10 jobs in the background, i usually want to wait > for the first one to exit so i can queue up another one, not wait for all of > them. If you set -m you can trap on SIGCHLD while waiting. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Re: Parallelism a la make -j / GNU parallel
On Friday 04 May 2012 13:46:32 Andreas Schwab wrote:
> Mike Frysinger writes:
> > i wish there was a way to use `wait` that didn't block until all the pids
> > returned. maybe a dedicated option, or a shopt to enable this, or a new
> > command.
> >
> > for example, if i launched 10 jobs in the background, i usually want to
> > wait for the first one to exit so i can queue up another one, not wait
> > for all of them.
>
> If you set -m you can trap on SIGCHLD while waiting.
awesome, that's a good mitigation
#!/bin/bash
set -m
cnt=0
trap ': $(( --cnt ))' SIGCHLD
for n in {0..20} ; do
(
d=$(( RANDOM % 10 ))
echo $n sleeping $d
sleep $d
) &
: $(( ++cnt ))
if [[ ${cnt} -ge 10 ]] ; then
echo going to wait
wait
fi
done
trap - SIGCHLD
wait
it might be a little racy (wrt checking cnt >= 10 and then doing a wait), but
this is good enough for some things. it does lose visibility into which pids
are live vs reaped, and their exit status, but i more often don't care about
that ...
-mike
signature.asc
Description: This is a digitally signed message part.
Re: Parallelism a la make -j / GNU parallel
Am 04.05.2012 20:53, schrieb Mike Frysinger:
> On Friday 04 May 2012 13:46:32 Andreas Schwab wrote:
>> Mike Frysinger writes:
>>> i wish there was a way to use `wait` that didn't block until all the pids
>>> returned. maybe a dedicated option, or a shopt to enable this, or a new
>>> command.
>>>
>>> for example, if i launched 10 jobs in the background, i usually want to
>>> wait for the first one to exit so i can queue up another one, not wait
>>> for all of them.
>> If you set -m you can trap on SIGCHLD while waiting.
> awesome, that's a good mitigation
>
> #!/bin/bash
> set -m
> cnt=0
> trap ': $(( --cnt ))' SIGCHLD
> for n in {0..20} ; do
> (
> d=$(( RANDOM % 10 ))
> echo $n sleeping $d
> sleep $d
> ) &
> : $(( ++cnt ))
> if [[ ${cnt} -ge 10 ]] ; then
> echo going to wait
> wait
> fi
> done
> trap - SIGCHLD
> wait
>
> it might be a little racy (wrt checking cnt >= 10 and then doing a wait), but
> this is good enough for some things. it does lose visibility into which pids
> are live vs reaped, and their exit status, but i more often don't care about
> that ...
> -mike
That won't work I don't think.
I think you meant something more like this?
set -m
cnt=0
trap ': $(( --cnt ))' SIGCHLD
set -- {0..20}
while [ $# -gt 0 ]; do
if [[ ${cnt} -lt 10 ]] ; then
(
d=$(( RANDOM % 10 ))
echo $n sleeping $d
sleep $d
) &
: $(( ++cnt ))
shift
fi
echo going to wait
sleep 1
done
which is basically what I did in my earlier example except I used USR2
instead of SIGCHLD and put it in a function to make it easier to use.
Re: Parallelism a la make -j / GNU parallel
On Fri, May 04, 2012 at 09:02:27PM +0200, John Kearney wrote:
> set -m
> cnt=0
> trap ': $(( --cnt ))' SIGCHLD
> set -- {0..20}
> while [ $# -gt 0 ]; do
> if [[ ${cnt} -lt 10 ]] ; then
>
> (
> d=$(( RANDOM % 10 ))
> echo $n sleeping $d
> sleep $d
> ) &
> : $(( ++cnt ))
> shift
> fi
> echo going to wait
> sleep 1
> done
You're busy-looping with a 1-second sleep instead of using wait and the
signal handler, which was the whole purpose of the previous example (and
of the set -m that you kept in yours). And $n should probably be $1 there.
Re: Parallelism a la make -j / GNU parallel
On Friday 04 May 2012 15:02:27 John Kearney wrote:
> Am 04.05.2012 20:53, schrieb Mike Frysinger:
> > On Friday 04 May 2012 13:46:32 Andreas Schwab wrote:
> >> Mike Frysinger writes:
> >>> i wish there was a way to use `wait` that didn't block until all the
> >>> pids returned. maybe a dedicated option, or a shopt to enable this,
> >>> or a new command.
> >>>
> >>> for example, if i launched 10 jobs in the background, i usually want to
> >>> wait for the first one to exit so i can queue up another one, not wait
> >>> for all of them.
> >>
> >> If you set -m you can trap on SIGCHLD while waiting.
> >
> > awesome, that's a good mitigation
> >
> > #!/bin/bash
> > set -m
> > cnt=0
> > trap ': $(( --cnt ))' SIGCHLD
> > for n in {0..20} ; do
> >
> > (
> >
> > d=$(( RANDOM % 10 ))
> > echo $n sleeping $d
> > sleep $d
> >
> > ) &
> >
> > : $(( ++cnt ))
> >
> > if [[ ${cnt} -ge 10 ]] ; then
> >
> > echo going to wait
> > wait
> >
> > fi
> >
> > done
> > trap - SIGCHLD
> > wait
> >
> > it might be a little racy (wrt checking cnt >= 10 and then doing a wait),
> > but this is good enough for some things. it does lose visibility into
> > which pids are live vs reaped, and their exit status, but i more often
> > don't care about that ...
>
> That won't work I don't think.
seemed to work fine for me
> I think you meant something more like this?
no. i want to sleep the parent indefinitely and fork a child asap (hence the
`wait`), not busy wait with a one second delay. the `set -m` + SIGCHLD
interrupted the `wait` and allowed it to return.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: Parallelism a la make -j / GNU parallel
Am 04.05.2012 21:13, schrieb Mike Frysinger:
> On Friday 04 May 2012 15:02:27 John Kearney wrote:
>> Am 04.05.2012 20:53, schrieb Mike Frysinger:
>>> On Friday 04 May 2012 13:46:32 Andreas Schwab wrote:
Mike Frysinger writes:
> i wish there was a way to use `wait` that didn't block until all the
> pids returned. maybe a dedicated option, or a shopt to enable this,
> or a new command.
>
> for example, if i launched 10 jobs in the background, i usually want to
> wait for the first one to exit so i can queue up another one, not wait
> for all of them.
If you set -m you can trap on SIGCHLD while waiting.
>>> awesome, that's a good mitigation
>>>
>>> #!/bin/bash
>>> set -m
>>> cnt=0
>>> trap ': $(( --cnt ))' SIGCHLD
>>> for n in {0..20} ; do
>>>
>>> (
>>>
>>> d=$(( RANDOM % 10 ))
>>> echo $n sleeping $d
>>> sleep $d
>>>
>>> ) &
>>>
>>> : $(( ++cnt ))
>>>
>>> if [[ ${cnt} -ge 10 ]] ; then
>>>
>>> echo going to wait
>>> wait
>>>
>>> fi
>>>
>>> done
>>> trap - SIGCHLD
>>> wait
>>>
>>> it might be a little racy (wrt checking cnt >= 10 and then doing a wait),
>>> but this is good enough for some things. it does lose visibility into
>>> which pids are live vs reaped, and their exit status, but i more often
>>> don't care about that ...
>> That won't work I don't think.
> seemed to work fine for me
>
>> I think you meant something more like this?
> no. i want to sleep the parent indefinitely and fork a child asap (hence the
> `wait`), not busy wait with a one second delay. the `set -m` + SIGCHLD
> interrupted the `wait` and allowed it to return.
> -mike
The functionality of the code doesn't need SIGCHLD, it still waits till
all the 10 processes are finished before starting the next lot.
it only interrupts the wait to decrement the counter.
to do what your talking about you would have to start the new subprocess
in the SIGCHLD trap.
try this out it might make it clearer what I mean
set -m
cnt=0
trap ': $(( --cnt )); echo SIGCHLD' SIGCHLD
for n in {0..20} ; do
(
d=$(( RANDOM % 10 ))
echo $n sleeping $d
sleep $d
echo $n exiting $d
) &
: $(( ++cnt ))
if [[ ${cnt} -ge 10 ]] ; then
echo going to wait
wait
fi
done
trap - SIGCHLD
wait
Re: Parallelism a la make -j / GNU parallel
Am 04.05.2012 21:11, schrieb Greg Wooledge:
> On Fri, May 04, 2012 at 09:02:27PM +0200, John Kearney wrote:
>> set -m
>> cnt=0
>> trap ': $(( --cnt ))' SIGCHLD
>> set -- {0..20}
>> while [ $# -gt 0 ]; do
>> if [[ ${cnt} -lt 10 ]] ; then
>>
>> (
>> d=$(( RANDOM % 10 ))
>> echo $n sleeping $d
>> sleep $d
>> ) &
>> : $(( ++cnt ))
>> shift
>> fi
>> echo going to wait
>> sleep 1
>> done
> You're busy-looping with a 1-second sleep instead of using wait and the
> signal handler, which was the whole purpose of the previous example (and
> of the set -m that you kept in yours). And $n should probably be $1 there.
>
see my response to mike.
what you are thinking about is either what I suggested or something like
this
function TestProcess_22 {
local d=$(( RANDOM % 10 ))
echo $1 sleeping $d
sleep $d
echo $1 exiting $d
}
function trap_SIGCHLD {
echo "SIGCHLD";
if [ $cnt -gt 0 ]; then
: $(( --cnt ))
TestProcess_22 $cnt &
fi
}
set -m
cnt=20
maxJobCnt=10
trap 'trap_SIGCHLD' SIGCHLD
for (( x=0; x
Re: Parallelism a la make -j / GNU parallel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 5/4/12 2:53 PM, Mike Frysinger wrote: > it might be a little racy (wrt checking cnt >= 10 and then doing a wait), but > this is good enough for some things. it does lose visibility into which pids > are live vs reaped, and their exit status, but i more often don't care about > that ... What version of bash did you test this on? Bash-4.0 is a little different in how it treats the SIGCHLD trap. Would it be useful for bash to set a shell variable to the PID of the just- reaped process that caused the SIGCHLD trap? That way you could keep an array of PIDs and, if you wanted, use that variable to keep track of live and dead children. Chet - -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+kOT4ACgkQu1hp8GTqdKs0iACfdXujLn96piuZGbJaIVu22TZ0 x/EAoIZo01MQJTCsEGvU0zmDghEEPjrA =DOKm -END PGP SIGNATURE-
Re: Parallelism a la make -j / GNU parallel
On Friday 04 May 2012 16:17:02 Chet Ramey wrote: > On 5/4/12 2:53 PM, Mike Frysinger wrote: > > it might be a little racy (wrt checking cnt >= 10 and then doing a wait), > > but this is good enough for some things. it does lose visibility into > > which pids are live vs reaped, and their exit status, but i more often > > don't care about that ... > > What version of bash did you test this on? Bash-4.0 is a little different > in how it treats the SIGCHLD trap. bash-4.2_p28. wait returns 145 (which is SIGCHLD). > Would it be useful for bash to set a shell variable to the PID of the just- > reaped process that caused the SIGCHLD trap? That way you could keep an > array of PIDs and, if you wanted, use that variable to keep track of live > and dead children. we've got associative arrays now ... we could have one which contains all the relevant info: declare -A BASH_CHILD_STATUS=( ["pid"]=1234 ["status"]=1# WEXITSTATUS() ["signal"]=13 # WTERMSIG() ) makes it easy to add any other fields people might care about ... -mike signature.asc Description: This is a digitally signed message part.
Re: Parallelism a la make -j / GNU parallel
On Friday 04 May 2012 15:25:25 John Kearney wrote:
> Am 04.05.2012 21:13, schrieb Mike Frysinger:
> > On Friday 04 May 2012 15:02:27 John Kearney wrote:
> >> Am 04.05.2012 20:53, schrieb Mike Frysinger:
> >>> On Friday 04 May 2012 13:46:32 Andreas Schwab wrote:
> Mike Frysinger writes:
> > i wish there was a way to use `wait` that didn't block until all the
> > pids returned. maybe a dedicated option, or a shopt to enable this,
> > or a new command.
> >
> > for example, if i launched 10 jobs in the background, i usually want
> > to wait for the first one to exit so i can queue up another one, not
> > wait for all of them.
>
> If you set -m you can trap on SIGCHLD while waiting.
> >>>
> >>> awesome, that's a good mitigation
> >>>
> >>> #!/bin/bash
> >>> set -m
> >>> cnt=0
> >>> trap ': $(( --cnt ))' SIGCHLD
> >>> for n in {0..20} ; do
> >>> (
> >>> d=$(( RANDOM % 10 ))
> >>> echo $n sleeping $d
> >>> sleep $d
> >>> ) &
> >>> : $(( ++cnt ))
> >>> if [[ ${cnt} -ge 10 ]] ; then
> >>> echo going to wait
> >>> wait
> >>> fi
> >>> done
> >>> trap - SIGCHLD
> >>> wait
> >>>
> >>> it might be a little racy (wrt checking cnt >= 10 and then doing a
> >>> wait), but this is good enough for some things. it does lose
> >>> visibility into which pids are live vs reaped, and their exit status,
> >>> but i more often don't care about that ...
> >>
> >> That won't work I don't think.
> >
> > seemed to work fine for me
> >
> >> I think you meant something more like this?
> >
> > no. i want to sleep the parent indefinitely and fork a child asap (hence
> > the `wait`), not busy wait with a one second delay. the `set -m` +
> > SIGCHLD interrupted the `wait` and allowed it to return.
>
> The functionality of the code doesn't need SIGCHLD, it still waits till
> all the 10 processes are finished before starting the next lot.
not on my system it doesn't. maybe a difference in bash versions. as soon as
one process quits, the `wait` is interrupted, a new one is forked, and the
parent goes back to sleep until another child exits. if i don't `set -m`,
then i see what you describe -- the wait doesn't return until all 10 children
exit.
-mike
signature.asc
Description: This is a digitally signed message part.
Re: Parallelism a la make -j / GNU parallel
Mike Frysinger writes: > not on my system it doesn't. maybe a difference in bash versions. as soon > as > one process quits, the `wait` is interrupted, a new one is forked, and the > parent goes back to sleep until another child exits. if i don't `set -m`, > then i see what you describe -- the wait doesn't return until all 10 children > exit. (bash) Bash POSIX Mode:: 46. The arrival of `SIGCHLD' when a trap is set on `SIGCHLD' does not interrupt the `wait' builtin and cause it to return immediately. The trap command is run once for each child that exits. (I think the description is backwards.) Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
