Re: bash parallel build: make[1]: warning: -j16 forced in submake: resetting jobserver mode.
On 4/13/24 12:40 AM, Cedric Blancher wrote:
Good morning!
Building bash HEAD on Cygwin 3.5.3 and Debian Linux 11 in make
parallel mode issues a warning, which looks like a Makefile bug:
$ make -h 16
...
make[1]: warning: -j16 forced in submake: resetting jobserver mode.
...
Can this be fixed please?
That's a message from `make'. What would you like to fix? It gets passed
down to subdirectory makes as part of ${MFLAGS}, and the parent and child
make instances cooperate to limit the number of parallel make processes.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
OpenPGP_signature.asc
Description: OpenPGP digital signature
Re: Parsing regression with for loop in case statement
On 4/10/24 8:51 PM, nbow...@draconx.ca wrote: Bash Version: 5.2 Patch Level: 21 Release Status: release Description: The POSIX shell grammar specifies that a newline may optionally appear before the in keyword of a for loop. [...] However, if the exact same loop is placed within the body of a case statement, current versions of bash exit with a syntax error: Thanks for the report. This came in in 2017 with better support for POSIX grammar rule 6, which is concerned with returning IN as the third word of a case or for command. The idea is that you need to keep track of whether or not you're returning an IN for a case command (in which case you enter a case pattern list state and can't perform alias expansion or return reserved words unless ESAC is the next token) or for a for command. The state management gets fooled by the multiple newlines; the parser needs more context. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Examples of concurrent coproc usage?
On 4/8/24 11:44 PM, Zachary Santer wrote: The fact that the current implementation allows the coproc fds to get into process substitutions is a little weird to me. A process substitution, in combination with exec, is kind of the one other way to communicate with background processes through fds without using FIFOs. I still have to close the coproc fds there myself, right now. So are you advocating for the shell to close coproc file descriptors when forking children for command substitutions, process substitutions, and subshells, in addition to additional coprocs? Right now, it closes coproc file descriptors when forking subshells. Consider the following situation: I've got different kinds of background processes going on, and I've got fds exec'd from process substitutions, fds from coprocs, If you have more than one coproc, you have to manage all this yourself already. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Examples of concurrent coproc usage?
On 4/9/24 10:46 AM, Zachary Santer wrote: If you want two processes to communicate (really three), you might want to build with the multiple coproc support and use the shell as the arbiter. If you've written a script for other people than just yourself, expecting all of them to build their own bash install with a non-default preprocessor directive is pretty unreasonable. This all started because I wasn't comfortable with the amount of testing the multiple coprocs code had undergone. If we can get more people to test these features, there's a better chance of making it the default. The part that I've been missing this whole time is that using exec with the fds provided by the coproc keyword is actually a complete solution for my use case, if I'm willing to close all the resultant fds myself in background processes where I don't want them to go. Which I am. Good deal. Whether the coproc fds should be automatically kept out of most kinds of subshells, like it is now; or out of more kinds than currently; is kind of beside the point to me now. Sure, but it's the potential for deadlock that we're trying to reduce. But, having a builtin to ensure the same behavior is applied to any arbitrary fd might be useful to people, especially if those fds get removed from process substitutions as well. What does this mean? What kind of builtin? And what `same behavior'? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/ OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Examples of concurrent coproc usage?
On 4/9/24 11:58 AM, Carl Edquist wrote:
On 4/4/24 7:23 PM, Martin D Kealey wrote:
I'm somewhat uneasy about having coprocs inaccessible to each other. I
can foresee reasonable cases where I'd want a coproc to utilize one or
more other coprocs.
In particular, I can see cases where a coproc is written to by one
process, and read from by another.
Can we at least have the auto-close behaviour be made optional, so that
it can be turned off when we want to do something more sophisticated?
With support for multiple coprocs, auto-closing the fds to other coprocs
when creating new ones is important in order to avoid deadlocks.
But if you're willing to take on management of those coproc fds yourself,
you can expose them to new coprocs by making your own copies with exec
redirections.
But this only "kind of" works, because for some reason bash seems to close
all pipe fds for external commands in coprocs, even the ones that the user
explicitly copies with exec redirections.
(More on that in a bit.)
On Mon, 8 Apr 2024, Chet Ramey wrote:
On 4/4/24 7:23 PM, Martin D Kealey wrote:
I'm somewhat uneasy about having coprocs inaccessible to each other. I
can foresee reasonable cases where I'd want a coproc to utilize one or
more other coprocs.
That's not the intended purpose,
The original intent was to allow the shell to drive a long-running process
that ran more-or-less in parallel with it. Look at examples/scripts/bcalc
for an example of that kind of use.
For what it's worth, my experience is that coprocesses in bash (rigged up
by means other than the coproc keyword) become very fun and interesting
when you allow for the possibility of communication between coprocesses.
(Most of my use cases for coprocesses fall under this category, actually.)
Sure, as long as you're willing to take on file descriptor management
yourself. I just don't want to make it a new requirement, since it's never
been one before.
Now, if you built bash with multiple coproc support, I would have expected
you could still rig this up, by doing the redirection work explicitly
yourself. Something like this:
coproc UP { stdbuf -oL tr a-z A-Z; }
coproc DOWN { stdbuf -oL tr A-Z a-z; }
# make user-managed backup copies of coproc fds
exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}
coproc THREEWAY { tee /dev/fd/$up_w /dev/fd/$down_w; }
But the above doesn't actually work, as it seems that the coproc shell
(THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even the
user-managed ones explicitly copied with exec.
File descriptors the user saves with exec redirections beyond [0-2]
are set to close-on-exec. POSIX makes that behavior unspecified, but
bash has always done it.
Shells don't offer any standard way to modify the state of that flag,
but there is the `fdflags' loadable builtin you can experiment with
to change close-on-exec.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
OpenPGP_signature.asc
Description: OpenPGP digital signature
Re: Exporting functions does not expand aliases in subshells
On 4/11/24 11:51 AM, Robert Elz wrote:
For how aliases can mess things up, with the bash way of parsing
command substitutions, if we do:
foo() { echo hello; X=$(echo goodbye); echo "$X"; }
and just run foo then we get "hello", "goodbye" (on successive lines).
Let's assume that this is an interactive shell, not in posix mode, since
those make a difference.
If we then do
alias echo='echo say'
and run foo again (without redefining it) then we get "hello" "say goodbye"
as the first "echo" was already parsed, the (later) alias can't
affect that one, but the one in the command substitution hasn't
been (properly) parsed yet, so the alias affects that.
In posix mode, bash does the same thing as ash-based shells.
As has come up several times before, the ad-hoc way bash used to parse
$()-style command substitutions before bash-5.2 -- which was not POSIX-
conformant, but still needs to be supported for backwards compatibility --
drives the difference in behavior.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
OpenPGP_signature.asc
Description: OpenPGP digital signature
Re: Examples of concurrent coproc usage?
Date:Sat, 13 Apr 2024 16:10:26 -0400 From:Chet Ramey Message-ID: <32bd9e76-24bc-4206-aa8a-8bcc81722...@case.edu> | File descriptors the user saves with exec redirections beyond [0-2] | are set to close-on-exec. POSIX makes that behavior unspecified, but | bash has always done it. It is the right way in general, it allows script fragments (like functions, loaded from external sources) to run external utilities without passing through file descriptors that they've never heard of, and so can't explicitly close. | Shells don't offer any standard way to modify the state of that flag, They don't.But in addition to fdflags our shell (not bash, and it has no coprocs, or anything similar) interprets the otherwise meaningless n>& n (or n<&n which is essentially the same thing) as a request to turn off close-on-exec for fd n. That can be a redirect on a specific command and applies (like any other redirect) just to that command (that command will have fd n open after it is exec'd), or it can be applied using "exec" to the fd in the shell itself (including in a later redirection in the same exec command which opens fd n originally, if the order of eval of the necessary bits ends up being defined - gets a bit messy when var expansions are needed). Apart from fdflags we don't have a mechanism to turn close-on-exec on again if it gets disabled that way, that's never seemed necessary. Perhaps you might have bash interpret it the same way - eventually perhaps it might become widespread enough to make it into the standard. And once there is a standard way to turn off close-on-exec maybe any shells which don't make that the default for fd's > 2 might see the advantages of doing it, and then perhaps that unspecified in the standard could go away too (sometime in the future). kre
Re: bash parallel build: make[1]: warning: -j16 forced in submake: resetting jobserver mode.
On Sat, Apr 13, 2024 at 12:41 AM Cedric Blancher wrote: > > Good morning! > > Building bash HEAD on Cygwin 3.5.3 and Debian Linux 11 in make > parallel mode issues a warning, which looks like a Makefile bug: > $ make -h 16 > ... > make[1]: warning: -j16 forced in submake: resetting jobserver mode. > ... > Can this be fixed please? It takes a certain makefile setting to cause this warning. E.g. -j16 can be specified in the makefile on the recipe line like this $ ls lib makefile $ cat makefile all:; $(MAKE) -C lib -j4 $ cat lib/makefile all:; $ make -j2 make -C lib -j4 make[1]: warning: -j4 forced in submake: resetting jobserver mode. make[1]: Entering directory '/home/dgoncharov/src/gmake/test/jobserver_reset/lib' make[1]: 'all' is up to date. make[1]: Leaving directory '/home/dgoncharov/src/gmake/test/jobserver_reset/lib' $ Submake detects that it has its own -j switch and it no longer participates in the parent make job server. Instead, the submake will become a master of its own jobserver and run up to 4 (in this example) jobs, and the parent make keep running its own job server with up to 2 (in this example) jobs. This can also be caused by settings in submake env. See https://www.gnu.org/software/make/manual/html_node/Error-Messages.html. You can check what env the submake in question has and which recipe runs it and see where this setting comes from. regards, Dmitry
Re: Examples of concurrent coproc usage?
On Sat, Apr 13, 2024 at 2:45 PM Chet Ramey wrote:
>
> On 4/8/24 11:44 PM, Zachary Santer wrote:
>
> > The fact that the current implementation allows the coproc fds to get
> > into process substitutions is a little weird to me. A process
> > substitution, in combination with exec, is kind of the one other way
> > to communicate with background processes through fds without using
> > FIFOs. I still have to close the coproc fds there myself, right now.
>
> So are you advocating for the shell to close coproc file descriptors
> when forking children for command substitutions, process substitutions,
> and subshells, in addition to additional coprocs? Right now, it closes
> coproc file descriptors when forking subshells.
Yes. I couldn't come up with a way that letting the coproc fds into
command substitutions could cause a problem, in the same sense that
letting them into regular ( ) subshells doesn't seem like a problem.
That bit is at least good for my arbitrary standard of "consistency,"
though.
At least in my use case, trying to use the coproc file descriptors
directly in a pipeline forced the use of a process substitution,
because I needed the coproc fds accessible in the second segment of
what would've been a three-segment pipeline. (Obviously, I'm using
'shopt -s lastpipe' here.) I ultimately chose to do 'exec {fd}> >(
command )' and redirect from one command within the second segment
into ${fd} instead of ending the second segment with '> >( command );
wait "${?}"'. In the first case, you have all the same drawbacks as
allowing the coproc fds into a subshell forked with &. In the second
case, it's effectively the same as allowing the coproc fds into the
segments of a pipeline that become subshells. I guess that would be a
concern if the segment of the pipeline in the parent shell closes the
fds to the coproc while the pipeline is still executing. That seems
like an odd thing to do, but okay.
Now that I've got my own fds that I'm managing myself, I've turned
that bit of code into a plain, three-segment pipeline, at least for
now.
> > Consider the following situation: I've got different kinds of
> > background processes going on, and I've got fds exec'd from process
> > substitutions, fds from coprocs,
>
> If you have more than one coproc, you have to manage all this yourself
> already.
Not if we manage to convince you to turn MULTIPLE_COPROCS=1 on by
default. Or if someone builds bash that way for themselves.
On Sat, Apr 13, 2024 at 2:51 PM Chet Ramey wrote:
>
> On 4/9/24 10:46 AM, Zachary Santer wrote:
>
> >> If you want two processes to communicate (really three), you might want
> >> to build with the multiple coproc support and use the shell as the
> >> arbiter.
> >
> > If you've written a script for other people than just yourself,
> > expecting all of them to build their own bash install with a
> > non-default preprocessor directive is pretty unreasonable.
>
> This all started because I wasn't comfortable with the amount of testing
> the multiple coprocs code had undergone. If we can get more people to
> test these features, there's a better chance of making it the default.
>
> > The part that I've been missing this whole time is that using exec
> > with the fds provided by the coproc keyword is actually a complete
> > solution for my use case, if I'm willing to close all the resultant
> > fds myself in background processes where I don't want them to go.
> > Which I am.
>
> Good deal.
>
> > Whether the coproc fds should be automatically kept out of most kinds
> > of subshells, like it is now; or out of more kinds than currently; is
> > kind of beside the point to me now.
>
> Sure, but it's the potential for deadlock that we're trying to reduce.
I hesitate to say to just set MULTIPLE_COPROCS=1 free and wait for
people to complain. I'm stunned at my luck in getting Carl Edquist's
attention directed at this. Hopefully there are other people who
aren't subscribed to this email list who are interested in using this
functionality, if it becomes more fully implemented.
> > But, having a builtin to ensure
> > the same behavior is applied to any arbitrary fd might be useful to
> > people, especially if those fds get removed from process substitutions
> > as well.
>
> What does this mean? What kind of builtin? And what `same behavior'?
Let's say it's called 'nosub', and takes fd arguments. It would make
the shell take responsibility for keeping those fds out of subshells.
Perhaps it could take a -u flag, to make it stop keeping the fd
arguments out of subshells. That would be a potential way to get bash
to quit closing coproc fds in subshells, as the user is declaring that
s/he is now responsible for those fds. Still a separate matter from
whether those fds get closed automatically in the parent shell at any
given point.
People who exec fds have to know to take responsibility for everywhere
they go, right now. Exec'ing fds like this is bound to be more common
than using coprocs, as is, I assume, e
