feature request: "wait --free-slot" for poor man's parallelization
Here's a bash feature I'd love to see, but don't have time to implement myself: a "--free-slot" flag to 'wait' that will wait until there is at least one free "slot" available, where a slot is basically a CPU core. Example usage: $ for ((n=0; n<100; n++)); do my_experiment $n > $n.out & wait --free-slot 4 done which I would run on a quad-CPU host. The meaning of the wait is: if I already have four background invocations of "my_experiment" running, wait until one finishes, then proceed. (If there are fewer than four running, proceed immediately.) This provides a nice sloppy way of letting me parallelize in a script without having to complicate things. "make -j" and "xargs -P" provide some of this capability, but are a lot more work to set up--I'd really like to have this at my fingertips for interactive stuff. Thanks, Mike
Re: feature request: "wait --free-slot" for poor man's parallelization
On Fri, Oct 3, 2008 at 4:13 AM, Jan Schampera <[EMAIL PROTECTED]> wrote: > Mike Coleman wrote: >> >> Here's a bash feature I'd love to see, but don't have time to >> implement myself: a "--free-slot" flag to 'wait' that will wait until >> there is at least one free "slot" available, where a slot is basically >> a CPU core. >> >> Example usage: >> >> $ for ((n=0; n<100; n++)); do >> my_experiment $n > $n.out & >> wait --free-slot 4 >> done >> >> which I would run on a quad-CPU host. The meaning of the wait is: if >> I already have four background invocations of "my_experiment" running, >> wait until one finishes, then proceed. (If there are fewer than four >> running, proceed immediately.) >> >> This provides a nice sloppy way of letting me parallelize in a script >> without having to complicate things. "make -j" and "xargs -P" provide >> some of this capability, but are a lot more work to set up--I'd really >> like to have this at my fingertips for interactive stuff. > > I did something like that as a script, using an array of PIDs and a big loop > and so on. I don't think it needs to be part of internal Bash features, it > wasn't that much work. I'm sure this could be done with a script, but I'm imagining a scenario like the above where I don't need to get out an editor or anything--it'd be a quick-and-dirty thing I could type in at the command line. If there's a way to write "wait-free-slot" as a function (that could be source'd in), I'm all ears, but I don't see it. I guess it could poll the current children with ps, but that seems kind of ugly. Mike
Re: feature request: "wait --free-slot" for poor man's parallelization
On Sat, Oct 4, 2008 at 7:02 PM, Bob Proulx <[EMAIL PROTECTED]> wrote: > If you start working with compute queues you will find that there are > endless different ways that people want to define job slots. It isn't > a simple problem. Sure, but for the scenario I have in mind, perfection is not necessary. What I'm looking for is a way to utilize my two (or four) cores for "do it right now" tasks. It doesn't really matter if I'm actually running four or eight processes--the situation I'm trying to avoid is starting up 400, which will swamp my machine. As for coding it up in a library, yes, that would be possible. This isn't nearly as useful as having it under my fingers on every Linux box I ever encounter (which would happen if the option were added to wait). It's kind of like the 'watch' command--very handy, but not worth the time to install if it's not present on a machine you encounter. Obviously, since I'm not offering to code this up, my opinion carries very little weight. I was hoping someone might decide they'd like the feature, too, and add it. (I suspect that it would actually be a very small change for someone who knows the bash code.) Regards, Mike
feature-request: brief syntax for $(type -p somecommand)
It would be nice if there was some really brief syntax for $(type -p somecommand)
Re: feature-request: brief syntax for $(type -p somecommand)
[Oops--I sent that incomplete.] It would be nice if there was some really brief syntax for $(type -p somecommand) I find myself using this all day long with 'ls', 'file', 'ldd', 'strings', 'nm', etc., and the current incantation is just long enough to be annoying. Mike
Re: feature-request: brief syntax for $(type -p somecommand)
On Thu, Apr 2, 2009 at 11:33 AM, Chris F.A. Johnson wrote:
> On Thu, 2 Apr 2009, Mike Coleman wrote:
>
>> [Oops--I sent that incomplete.]
>>
>> It would be nice if there was some really brief syntax for
>>
>> $(type -p somecommand)
>>
>> I find myself using this all day long with 'ls', 'file', 'ldd',
>> 'strings', 'nm', etc., and the current incantation is just long enough
>> to be annoying.
>
> Use a function, e.g.:
>
> p()
> {
> pp=$( type -p "$@" )
> }
Hmm. So I would use this like so?
$ p somecommand; ls -l $pp
I guess it's not obvious that that's an improvement. Along these
lines, though, is there some way to do something like this
script named 'p':
$!/bin/bash
exec type -p "$1"
which could then be used like
$ ls -l $(p somecommand)
except without the external script?
Re: feature-request: brief syntax for $(type -p somecommand)
On Thu, Apr 2, 2009 at 3:16 PM, Jan Schampera wrote: > Mike Coleman wrote: >> [Oops--I sent that incomplete.] >> >> It would be nice if there was some really brief syntax for >> >> $(type -p somecommand) >> >> I find myself using this all day long with 'ls', 'file', 'ldd', >> 'strings', 'nm', etc., and the current incantation is just long enough >> to be annoying. >> >> Mike >> >> > > Why do you need to get the path of a program that's in PATH? Well, for example, I may have an undocumented program 'foobar' in my path, and running 'strings' on it is a good way to grab its usage message and/or look at its embedded messages, etc. I might run 'ldd' on it to see what libs it's depending on, 'ls' to see when it was last updated, who owns it, etc., 'file' to see what it is (script or executable, 32-bit or 64, etc.). Obviously for all of these, I could do a 'type -p' first and then paste the result in as the argument for these other commands, but it's handy not to have to do that. Currently I have a whole bunch of aliases (pls, pfile, pldd, pstrings, etc.) to do this, but it seems like a useful enough pattern that it'd be worth having a general shortcut for it. Mike
Re: feature-request: brief syntax for $(type -p somecommand)
On Thu, Apr 2, 2009 at 11:57 AM, Chris F.A. Johnson wrote:
> If that's what you want, you can include it in the function:
>
> p()
> {
> _p=$( type -p "$@" )
> [ -n "$_p" ] && ls -l $_p
> }
This is more or less what I'm doing now, with one function for each
command. I suppose the command could be an argument, too, so that one
could run
$ p ls somecommand
$ p ldd somecommand
$ p strings somecommand
$ p file somecommand
$ p nm somecommand
though there's no command completion this way. I guess I could add that, too.
One problem, though, like the "wait for slot" feature I suggested a
few months ago, is that something like this is quite a bit more useful
when it's available everywhere (without having to drag one's bash
library around).
Mike
bash treats SIGSTOP in child process as child termination?
I notice that if I do this in one (interactive) shell $ for n in 1 2 3 4 5; do /bin/sleep 60; echo $n; done and then 'kill -STOP' the sleep process (from another window), that bash will proceed to run the next loop iteration. That is, it echos '1' and starts a new /bin/sleep, even while the first one is sitting there stopped. (This doesn't appear to be specific to 'sleep'--an example with 'dd' also does the same thing.) It seems to me that this loop should just wait until the process is 'kill -CONT'ed and keep right on going as if nothing had happened. Is there any reason not to do this? Mike
Re: bash treats SIGSTOP in child process as child termination?
On Tue, Aug 4, 2009 at 12:44 PM, Chet Ramey wrote: >> It seems to me that this loop should just wait until the process is >> 'kill -CONT'ed and keep right on going as if nothing had happened. Is >> there any reason not to do this? > > Ummm...yes. It renders job control useless. If we have the shell > hang until a stopped child process is continued, why run with job > control at all? If you want to treat the entire loop as a stoppable > unit, run it in a subshell. I hadn't thought of that. And, indeed, just surrounding the loop with parentheses, giving a subshell, does solve my problem. (Well, except that the users typing in these loops are novices, so now I have the problem of trying to get them to surround these loops with parentheses. :-) I'm still not sure that the original behavior makes sense, as opposed to simply hanging until the child is continued (with the hang interruptible by Control-C). Is there some use case in which this provides a benefit? It surprised me, and I can't recall ever having seen it documented. This scenario is not something that will happen accidentally, since there's really no way to SIGSTOP the child without doing it from another shell, so the prospect of a user ending up in front of a "hung" shell doesn't seem like that much of a problem. Here's my use case, for what it's worth: Very novice users use these loops to run series of jobs, each of which may take hours. It would be handy for me in an admin role to occasionally be able to STOP/CONT these jobs as a method of ad hoc priority shuffling (to make one job step in front of another). I can't do this, though, if this breaks their loops. Mike
