Re: Examples of concurrent coproc usage?
Some other way of thinking:
> On 3/14/24 5:58 AM, Carl Edquist wrote:
> > ...
> > But if you try...
> >
> > $ coproc WC { wc; }
> > $ coproc CAT { cat; }
> > $ exec {WC[1]}>&-
> > $ read -u ${WC[0]} X
> >
> > # HANGS
To prevent `exec {WC[1]}>&-` to close both FD, just because subpid is
ended, I open a *backup FD*:
coTest() {
local X WC CAT bakWcOut wcPid
coproc WC { exec wc; }
wcPid=$!
coproc CAT { exec cat; }
echo foo exec bar >&${WC[1]} # Not really usefull for demo
exec {bakWcOut}>&${WC[0]}
exec {WC[1]}>&-
wait $wcPid
read -u ${bakWcOut} X
echo $X
exec {bkWc}>&-
exec {CAT[1]}>&-
wait
ls -l /dev/fd/
}
This function's output look good (on my Debian's bash 5.2.15(1)-release):
$ coTest
[1] 1606
bash: warning: execute_coproc: coproc [1606:WC] still exists
[2] 1607
[1]- Donecoproc WC { exec wc; }
1 3 13
[2]+ Donecoproc CAT { exec cat; }
total 0
lrwx-- 1 user user 64 Apr 17 11:12 0 -> /dev/pts/3
lrwx-- 1 user user 64 Apr 17 11:12 1 -> /dev/pts/3
lrwx-- 1 user user 64 Apr 17 11:12 2 -> /dev/pts/3
lr-x-- 1 user user 64 Apr 17 11:12 3 -> /proc/1608/fd
--
Félix Hauri -- http://www.f-hauri.ch
Re: Examples of concurrent coproc usage?
On 4/16/24 2:46 AM, Carl Edquist wrote: But the shell is pretty slow when you ask it to shovel data around like this. The 'read' builtin, for instance, cautiously does read(2) calls of a single byte at a time. It has to do it that way to find the delimiter on a non-seekable file descriptor, since it has to leave everything it didn't consume available on stdin. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Examples of concurrent coproc usage?
On 4/15/24 1:01 PM, Carl Edquist wrote: On Sat, 13 Apr 2024, Chet Ramey wrote: The original intent was to allow the shell to drive a long-running process that ran more-or-less in parallel with it. Look at examples/scripts/bcalc for an example of that kind of use. Thanks for mentioning this example. As you understand, this model use case does not require closing the coproc fds when finished, because they will be closed implicitly when the shell exits. (As bcalc itself admits.) And if the coproc is left open for the lifetime of the shell, the alternate behavior of deferring the coproc deallocation (until both coproc fds are closed) would not require anything extra from the user. The bcalc example does close both coproc fds though - both at the end, and whenever it resets. And so in this example (which as you say, was the original intent), the user is already explicitly closing both coproc fds explicitly; so the alternate deferring behavior would not require anything extra from the user here either. ... Yet another point brought to light by the bcalc example relates to the coproc pid variable. The reset() function first closes the coproc pipe fds, then sleeps for a second to give the BC coproc some time to finish. An alternative might be to 'wait' for the coproc to finish (likely faster than sleeping for a second). If the coproc has some problem and doesn't exit immediately, `wait' without options will hang. That's why I opted for the sleep/kill-as-insurance combo. (And before you ask why I didn't use `wait -n', I wrote bcalc in 30 minutes after someone asked me a question about doing floating point math with awk in a shell script, and it worked.) -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Examples of concurrent coproc usage?
On 3/14/24 5:58 AM, Carl Edquist wrote: Separately, I consider the following coproc behavior to be weird, fragile, and broken. Yes, I agree that coprocs should survive being suspended. The most recent devel branch push has code to prevent the coproc being reaped if it's stopped and not terminated. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Examples of concurrent coproc usage?
On Wed, 17 Apr 2024, Chet Ramey wrote:
> On 4/16/24 2:46 AM, Carl Edquist wrote:
>
> > But the shell is pretty slow when you ask it to shovel data around like
> > this. The 'read' builtin, for instance, cautiously does read(2) calls of a
> > single byte at a time.
>
> It has to do it that way to find the delimiter on a non-seekable file
> descriptor, since it has to leave everything it didn't consume available
> on stdin.
Has anyone tried asking any of the kernel teams (Linux, BSD, or other) to
add a new system call such as readln() or readd()?
I envisage this working like stty cooked mode works on a tty, except it
would also work on files, pipes, and sockets: you'd get back *at most* as
many bytes as you ask for, but you may get fewer if a delimiter is found.
The delimiter is consumed (and returned in the buffer), but everything
following a delimiter is left available for a subsequent read.
For a tty, code like
tc_setattr(fd, &(struct termios){ .c_iflags = ICANON, .c_cc = { [EOL] = '\n'
}});
ssize_t n = read(fd, &buf, sizeof buf);
could become just
ssize_t n = readd(fd, &buf, sizeof buf, "\n", 1, 0 /*flags*/);
or perhaps even
ssize_t n = readdv(fd,
&buf, sizeof buf,
(struct iovec[]){{"\n", 1}}, 1,
0 /*flags*/);
I'm not sure whether multi-byte delimiters should be allowed, as it's
unclear what to do when you get an incomplete delimiter at the end of the
buffer, but an iovec interface would at least allow that as a future
possibility.
As Linux kernel developers have found, it's better to *always* include a
flags argument, even if you can't think of a use for it yet; but in this
case O_NONBLOCK and O_PEEK could immediately be useful.
-Martin
PS: I initially wondered about having
ssize_t n = readvdv(fd,
(struct iovec[]){{&buf, sizeof buf}}, 1,
(struct iovec[]){{"\n", 1}}, 1);
but a vectored read isn't much of a saving unless the overall size is very
large, and enormous reads probably shouldn't use this facility since the
in-kernel byte scan would potentially block reads and writes by other
processes.
For the same reason I would allow the read size to be silently capped at a
value chosen by the kernel, probably a small multiple (1?) of the IO block
size or the pipe buffer size or the tty input buffer size.
