Re: "builtin jobs" does not output to stdout.
On 2/11/23 8:54 AM, 岡部将幸 wrote: Bash Version: 5.2 Patch Level: 15 Release Status: release Description: builtin command "jobs" behaves differntly depending on the presence or absence of "builtin". "jobs" outputs to stdout, but "builtin jobs" does not output to stdout. Repeat-By: "sleep 1000 &" "builtin jobs | cat" -> no output "jobs | cat" -> [1]+ Running Yes. As described in later messages, there is a special case that allows the output of `jobs' to be piped. This came in at some point between bash-1.05 (March 1990) and bash-1.08 (May 1991). I never extended this special case to `builtin' or `command'. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: "builtin jobs" does not output to stdout.
On 2/11/23 11:14 AM, Koichi Murase wrote: 2023年2月12日(日) 0:42 Robert Elz : Why would you want to ever say "builtin jobs" though? The jobs command has to be buikt in to work. `jobs' can be overwritten by a shell function. For example, when a user wants to modify the behavior of `jobs' for interactive uses, a typical solution is to override `jobs' with a shell function and call `builtin jobs' in the overriding function. I guessed you have asked this because `jobs' would be specified as a special built-in utility [XCU 2.14], which cannot be hidden by a shell-function name, but it doesn't seem to be the case actually; There's no way `jobs' would ever be a special builtin. It's always been UPE shaded (User Portability). Job control wasn't even mandatory in POSIX until Issue 6 (2004), and the specification that jobs can be run when job control isn't enabled came well after that. The distinction between regular and special builtins has always seemed arbitrary -- special builtins are defined in terms of their effects, but the only rationale for why those particular ones were chosen is "this is what the Bourne shell did." Which is fine, but mostly an artifact of that implementation; no discussion of why or whether it's a good idea to carry that forward. `jobs' doesn't seem to be specified to be a special built-in utility. The 1988 Bourne shell didn't have it, so it can't be special. I naively think that it is a valid request that `builtin jobs' would behave in the same way as `jobs' in this context. POSIX also states that the results are unspecified when the command name matches `builtin' according to [XCU 2.9.1 / Command Search and Execution / rule 1b]. Whether or not `builtin' is specified by POSIX isn't really relevant. That list just means that some shells have implemented `builtin' as a builtin command, so portable applications should take care with it. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: "builtin jobs" does not output to stdout.
On 2/11/23 8:15 PM, Robert Elz wrote: Date:Sun, 12 Feb 2023 01:14:12 +0900 From:Koichi Murase Message-ID: | `jobs' can be overwritten by a shell function. For example, when a | user wants to modify the behavior of `jobs' for interactive uses, a | typical solution is to override `jobs' with a shell function and call | `builtin jobs' in the overriding function. The standard way to do that would be to use "command jobs" - but in either of those cases, it is possible that the way that the shell recognises the special case that allows the command in a sub-shell to output information from the parent shell, rather than the sub-shell's information may not be met. The same applies to the "trap" command which has similar issues. `command jobs' works (I had to remind myself that it does). It's another implementation artifact; it happened separately from the `jobs_hack'. It's because `command' runs the rest of the words through the execution path again, while `builtin' simply looks up and invokes a shell builtin directly. | I guessed you have asked this because `jobs' would be specified as a | special built-in utility [XCU 2.14], which cannot be hidden by a | shell-function name, but it doesn't seem to be the case actually; bash has very little belief in the concept of "special bui8lt-in". "Special builtin" is not a particularly useful concept, but bash embraces it in posix mode. Chet can decide if making bash handle more cases than it currently does is worth it or not for bash and its users. I think handling it in `jobs' and `command jobs' is enough for now. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: "builtin jobs" does not output to stdout.
On 2/13/23 6:43 AM, Koichi Murase wrote:
I guess just the support for ksh's ${ list; } [1] would make
everything simple and clear. One can simply call ${ jobs; }, ${ trap
-p; }, etc. without thinking about subshells.
[1] https://lists.gnu.org/archive/html/help-bash/2020-05/msg00077.html
The text about syntatic sugar still applies.
I haven't checked what POSIX says about the jobs in subshells,
POSIX requires that the shell execution environment include what is
essentially the jobs list;
https://www.austingroupbugs.net/view.php?id=1254
discusses this extensively. However,
"The jobs utility does not work as expected when it is operating in its own
utility execution environment because that environment has no applicable
jobs to manipulate."
but at
least Bash maintains a separate "job list" of a subshell (which is
accessible from the built-in command `jobs') regardless of whether the
"job control" is turned on or not.
POSIX requires this:
"The jobs utility is not dependent on the job control option, as are the
seemingly related bg and fg utilities because jobs is useful for examining
background jobs, regardless of the condition of job control."
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: "builtin jobs" does not output to stdout.
Chet Ramey writes: > Yes. As described in later messages, there is a special case that allows > the output of `jobs' to be piped. This came in at some point between > bash-1.05 (March 1990) and bash-1.08 (May 1991). I never extended this > special case to `builtin' or `command'. Is this (are these) special cases documented? I just looked through my (admittedly obsolete) bash man page and couldn't find any mention of special behavior of the "jobs" builtin. But presumably careful reading of the documentation of "command", "builtin", and "jobs" would show which cases work in which way. Dale
Re: "builtin jobs" does not output to stdout.
On 2/14/23 2:39 PM, Dale R. Worley wrote: Chet Ramey writes: Yes. As described in later messages, there is a special case that allows the output of `jobs' to be piped. This came in at some point between bash-1.05 (March 1990) and bash-1.08 (May 1991). I never extended this special case to `builtin' or `command'. Is this (are these) special cases documented? I just looked through my (admittedly obsolete) bash man page and couldn't find any mention of special behavior of the "jobs" builtin. But presumably careful reading of the documentation of "command", "builtin", and "jobs" would show which cases work in which way. Nothing special. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: number of bugs
There is a bug tracker at https://savannah.gnu.org/support/?group=bash. Dale
Re: Having an alias and a function with the same name leads to some sort of recursion
>> Looking at the manual page, it says >> >> ALIASES >>Aliases allow a string to be substituted for a word when it is used >> as >>the first word of a simple command. Martin suggested (but IIUC didn't sent to this list): > "Beginning of a simple command" should probably be replaced by something > more along the lines of "beginning of any command that does not start with > a keyword (such as "while", "if", "case", etc) or assignment. Though I think by "keyword" he means "reserved word". Dale
special built-in [Was: Re: "builtin jobs" does not output to stdout.]
Date:Tue, 14 Feb 2023 11:04:48 -0500
From:Chet Ramey
Message-ID: <8be447ca-5416-a64f-7e62-1e61102e8...@case.edu>
| "Special builtin" is not a particularly useful concept
I disagree - most of the time the difference doesn't matter much,
but that the special built-ins cause the shell to exit if they fail
means that script writers don't need to do things like
break || exit 99
just in case something is wrong with the script, and the break doesn't
happen in a loop, and so is an error. With it being a special built-in
the script simply exits at that point, otherwise if the script writer
hasn't tested for the error, the script just keeps running to the next
command, which is fairly clear is not what was intended.
That's somewhat esoteric, as things like break/continue/ ... rarely
ever fail.
But for "eval" it matters, consider
eval '| grep foo' || {
# code to handle what happens when foo isn't there
}
That's not likely as written, but could happen from
eval " $command | grep foo" || ...
if "$command" turns out to be unset or null at this point due to some
bug or other. That results in a syntax error from eval, which is a
special built-in failure, which should cause the shell to exit. Allowing
it to continue with just a failure status means the script believes that
the grep failed.
That is, I don't really think that (from an earlier message):
| the only rationale for why those particular ones were chosen
| is "this is what the Bourne shell did."
or perhaps stated better, yes, that is more or less why they were chosen, but
that makes it seem like some of the other (bizarre) things that the Bourne
shell did (probably as a code space saving mechanism), whereas I suspect
that this division was probably more carefully considered from the start.
The special built-in commands are commands that could almost be reserved
words, and often would be, but were implemented as commands instead (and
probably are better that way).
kre
Re: "builtin jobs" does not output to stdout.
2023年2月14日(火) 4:32 Oğuz :
> 13 Şubat 2023 Pazartesi tarihinde Koichi Murase
> yazdı:
>> One of the
>> cases that the fork cost of $() becomes a problem and that other
>> languages cannot be really used is the prompt setting `PS1' containing
>> command substitutions.
>
> Bash has many escape sequences to enrich your prompt strings. If they don't
> meet your needs and you find yourself embedding so many command substitutions
> in them that there is a discernible delay,
That is not my personal use case. In fact, my PS1 is just as simple as
PS1='[\[\]\u@\h\[\] \j \W]\$ '. Honestly, I
do more non-trivial and unusual stuff in my personal configurations,
but I'm personally satisfied with the current feature set of Bash
(with many supplementing shell functions) unless the long-lasting Bash
behaviors would be broken by the literal interpretation of a corner of
the standard that people haven't paid much attention historically.
I just naively think that ${ list; } would help normal users who
wouldn't like to do non-trivial stuff. There are many users who try to
embed multiple command substitutions to acquire git repository
information or other version-control-system information for the
current directory, the information of the current Python/Ruby virtual
environment, etc. The upstream Git itself prepares a utility intended
to be used in the command substitution embedded in PS1 as
/contrib/completion/git-prompt.sh in the git repository of git.
Embedding command substitutions for the VCS information and virtual
environment information in the prompt string is nothing wrong, and two
or three command substitutions are generally not considered "so many
command substitutions".
> perhaps it's time you give other shells a try. I hear fish is good.
For me, there are no other shells that are sufficiently matured for
daily uses and have essentially different languages and libraries that
are fully featured as Java, Python, C, etc.
You might hear of Fish shell is good, but which part of Fish shell is
considered good? They are just good for interactive behaviors. When we
focus on the language design, Fish shell is actually worse than Bash.
They tried to change superficial syntax like the keywords "esac",
"fi", etc., but they made a number of mistakes in the initial language
design, and many serious language defects are left unresolved for
compatibility. For example, it was impossible in Fish shell to pass
the multiline result of command substitutions as an argument to
another command until recently. They gave up and now have two
different command substitution syntaxes, () and $().
Instead, one might consider moving to other shells with languages
similar to POSIX sh or Bash, but in that case, the limitations of `sh'
languages still apply.
>> An example is a shell function: a shell
>> function can change the global variables (or previous-scope variables
>> with Bash's dynamic scoping) and, at the same time, can output data to
>> stdout. When one wants to capture the stdout of such a function while
>> keeping the changes made to the global variables, the command
>> substitution cannot be directly used.
>
>
> It's not a good example because there isn't much difference between `f() {
> list; }; x=$(f)' and `x=$(list)', they both read the output of `list' into
> `x'.
The assumption is the opposite. I'm not talking about converting
existing command substitutions to shell functions, of course. I'm
thinking the case where we first have a shell function that is not
necessarily intended for the command substitution but for general
uses, and then we would like to use its output in a particular place.
> When is it absolutely necessary to both retain the side effects of `list' and
> save its output to a variable?
Of course, there is no case where ${ list; } is "absolutely
necessary". But in that logic, the shell functions are not "absolutely
necessary", and even the command substitutions are not "absolutely
necessary" as everything can be in principle processed in combinations
of pipelines and { list; }, etc. The reason that we have shell
functions and command substitutions is that it makes it easier to
organize shell programs. For the same reason, functions are allowed to
return results with a side effect while it is not "absolutely
necessary" as one can rewrite the entire program in a purely
functional way. Then, when one wants to use the output of such a
function, it is useful to have ${ func; } while it is not "absolutely
necessary" because one can always reimplement the entire program so
that it doesn't have any functions with side effects.
>> Anyway, if there are no use cases or no benefits at all, ksh didn't
>> have the feature.
>
> Ksh has many features no one ever uses; some of them don't make any sense,
> some don't even work. Check out , it's a mess
> compared to bash.
OK, I understand what you try to say, (though I regard "no one ever"
and "any sense" as exaggerations; at least the d
Re: "builtin jobs" does not output to stdout.
2023年2月15日(水) 0:51 Chet Ramey : > On 2/11/23 11:14 AM, Koichi Murase wrote: > > 2023年2月12日(日) 0:42 Robert Elz : > >> Why would you want to ever say "builtin jobs" though? > >> The jobs command has to be buikt in to work. > > > > `jobs' can be overwritten by a shell function. For example, when a > > user wants to modify the behavior of `jobs' for interactive uses, a > > typical solution is to override `jobs' with a shell function and call > > `builtin jobs' in the overriding function. > > > > I guessed you have asked this because `jobs' would be specified as a > > special built-in utility [XCU 2.14], which cannot be hidden by a > > shell-function name, but it doesn't seem to be the case actually; > > There's no way `jobs' would ever be a special builtin. It's always been > UPE shaded (User Portability). Job control wasn't even mandatory in > POSIX until Issue 6 (2004), and the specification that jobs can be run > when job control isn't enabled came well after that. Thank you for the information. I find it interesting to hear about the historical background. The statement about the special built-in was just my initial guess for the reason why kre seemed to claim that `builtin jobs' is equivalent to `jobs', but I just checked the standard and immediately knew my guess was incorrect. I still don't think `builtin jobs' is equivalent to `jobs' because `jobs' can pick up a shell function. > > I naively think that it is a valid request that `builtin jobs' would > > behave in the same way as `jobs' in this context. POSIX also states > > that the results are unspecified when the command name matches > > `builtin' according to [XCU 2.9.1 / Command Search and Execution / > > rule 1b]. > > Whether or not `builtin' is specified by POSIX isn't really relevant. That > list just means that some shells have implemented `builtin' as a builtin > command, so portable applications should take care with it. I just meant that even if Bash treats `builtin jobs' the same as `jobs' in these particular subshell contexts, it doesn't conflict with POSIX. So, POSIX seems to be irrelevant when we think about the behavior when `builtin' is involved.
Re: "builtin jobs" does not output to stdout.
2023年2月15日(水) 1:20 Chet Ramey :
>
> On 2/13/23 6:43 AM, Koichi Murase wrote:
>
> > I guess just the support for ksh's ${ list; } [1] would make
> > everything simple and clear. One can simply call ${ jobs; }, ${ trap
> > -p; }, etc. without thinking about subshells.
> >
> > [1] https://lists.gnu.org/archive/html/help-bash/2020-05/msg00077.html
>
> The text about syntatic sugar still applies.
> > I haven't checked what POSIX says about the jobs in subshells,
>
> POSIX requires that the shell execution environment include what is
> essentially the jobs list;
Thank you for the information. This is what I wanted to find. Thank
you. (At that time, I just quickly searched for the job list, but I
couldn't find it. Also the definition of "a job" didn't seem to be
given in the standard, but I now found "Background Jobs" in XBD)
> https://www.austingroupbugs.net/view.php?id=1254
>
> discusses this extensively. However,
>
> "The jobs utility does not work as expected when it is operating in its own
> utility execution environment because that environment has no applicable
> jobs to manipulate."
>
>
> > but at
> > least Bash maintains a separate "job list" of a subshell (which is
> > accessible from the built-in command `jobs') regardless of whether the
> > "job control" is turned on or not.
>
> POSIX requires this:
>
> "The jobs utility is not dependent on the job control option, as are the
> seemingly related bg and fg utilities because jobs is useful for examining
> background jobs, regardless of the condition of job control."
Thank you for all the information.
--
Koichi
Re: Having an alias and a function with the same name leads to some sort of recursion
On Tue, Feb 14, 2023, 8:58 PM Dale R. Worley wrote: > >> Looking at the manual page, it says > >> > >> ALIASES > >>Aliases allow a string to be substituted for a word when it is > used as > >>the first word of a simple command. > > Martin suggested (but IIUC didn't sent to this list): > > "Beginning of a simple command" should probably be replaced by something > > more along the lines of "beginning of any command that does not start > with > > a keyword (such as "while", "if", "case", etc) or assignment. > > Though I think by "keyword" he means "reserved word". > i mix my maybe again-fail-in-place ( topic ) std root@localhost:~# alias if='if ' mid='[[ 1 ]] ' mid2='; then ' c1='printf ye\\n ' end='; fi ' root@localhost:~# if mid mid2 c1 end ye root@localhost:~# Dale > >
Re: Regression in pattern substitution with compat42
On Tue, 14 Feb 2023 at 01:00, Chet Ramey wrote: In this case, the behavior is controlled by an option. You just don't like > the default setting. > We had this exact same discussion back in November. > Too right I don't like it, and the previous discussion was wholly unsatisfactory. I do not understand why it is considered acceptable to add new features at the expense of breaking existing scripts when the breakage is so clearly unnecessary. (In this case, for example, a new micro-localised special parameter could have avoided breaking existing valid scripts, and as a side bonus, would have avoided re-parsing nested expansions.) I support one release at a time -- the current one. The distros do what > they do. My point is not to introduce a new development stream. Quite the contrary, I'm trying to narrow down what needs to be supported. As it stands, every release of Bash that's ever been made is equally "valid" for all time. Which means that after declaring some minimum version of Bash for our script, we're obliged to make it work on every subsequent version. I would like to stop adding to the burden for script writers. There are still Linux versions shipping with bash-4.2, which was released > in 2010. > Is anyone surprised by this, when new versions cannot be trusted? > I do not have the capacity to support all of those. > > > PS: the current "beta testing" arrangement seems to me to have too few > > participants by at least two orders of magnitude. We need to encourage > > "bleeding edge" Linux distros and similar to offer pre-built alpha and > beta > > releases of Bash. > > All the bash testing releases are public, and announced here and other > places. I don't have the time or energy to chase the distros urging them > to test releases -- it's hard enough to get the translation project to > react to testing releases. > I've been saying "we", not "Chet". We, the whole community, need to do this. > compat51 ... since it's a breaking change to previous behaviour? > > That's not what the compat mode is for. It's intended as a temporary > stopgap for things that change between releases -- bug fixes or whatever -- > when there's no other way to control that behavior, and gives people an > opportunity to fix their code. > If that's all that it's "for" then there's a gaping hole in the language that needs filling. How does a coder write a script now, and have some reasonable guarantee that it won't break when Bash 5.3 comes out? "Temporarily add «shopt -s compat52» when Bash 5.3 is released" is mind-numbingly wrong-headed. The "opportunity to fix their code" simply does not exist for a sizable proportion of scripts, where there's no formal distribution process, no maintainer, no support, and no coding competence in the end-user. The shell is a god-awful mis-designed language from almost every vantage point. The ONLY thing going for it is its putative stability. In languages like Perl or Go, all breaking changes need to be explicitly turned on, either by requesting individual features, or by declaring a minimum version that has all the required features. All other 'new' features remain turned off. If I can't convince you of the vital importance of this, then I give up... So if I make "Bash6" as a fork of Bash 5.1, with the following guarantees, would anyone be interested in adopting it? 1. Changes are only enabled when explicitly requested. (We don't guess ahead of time which changes will break something.) 2. An approved list LTS releases (selected from previous releases, not a separate development branch), and expiry dates for other (interim) releases. 3. Document expectations for distros, including: - Install with version numbers in filenames (same approach as used for ELF shared library numbering in Linux and elsewhere), with symlink for just "bash"; - Only install LTS releases in embedded devices or where the user cannot manage software updates; - Encourage users to install multiple version, especially all LTS versions; - If /bin/. exists in your distro, ensure that /bin/bash and /bin/bash$version are (or are symlinks to) the relevant versions. 4. Allow scripts to declare the minimum required Bash version (as command-line option so it can go in #!) 5. Allow scripts to declare the maximum tested Bash version (the "current version" when they're written). This would have a permanent effect, unlike the current shopt compat, which requires editing when one later discovers breakage caused by deployment of a future version of Bash. (Most likely I would just expand "shopt compat" by including "compatXX" in release XX, and not delay it until the next release.) 6. Implement new features as non-breaking changes where possible, or "least surprise" if not. New variables only in a predefined namespace (probably BASH_* but I'm open to suggestions at this point). Conversely, there are some ancient
Re: Regression in pattern substitution with compat42
On Wed, 15 Feb 2023 12:48:10 +1000 Martin D Kealey wrote: > > There are still Linux versions shipping with bash-4.2, which was released > > in 2010. > > > > Is anyone surprised by this, when new versions cannot be trusted? It is, perhaps, telling that two of the most notable rolling Linux distributions continue to hold off from issuing 5.2 in their stable channels. That includes Arch, the very same Arch that routinely performs same-day releases from upstream without any semblance of QA - including for glibc - and generally doesn't bother to backport any critical fixes between upstream releases. > So if I make "Bash6" as a fork of Bash 5.1, with the following guarantees, > would anyone be interested in adopting it? > It seems like a lot to take on but count me in as being interested. -- Kerin Millar
Re: "builtin jobs" does not output to stdout.
On 2/15/23 2:52 AM, Koichi Murase wrote:
two or three command substitutions are generally not considered "so
many command substitutions".
I can't reproduce a great deal of unresponsiveness with five or six
either, and my computer is very old too. I think this "delay" you
mentioned has more to do with the commands being substituted than bash.
You might hear of Fish shell is good, but which part of Fish shell is
considered good? They are just good for interactive behaviors. When we
focus on the language design, Fish shell is actually worse than Bash.
It's all about priorities; if your top priority is fashionable prompt
strings, you won't mind the inferior language.
Of course, there is no case where ${ list; } is "absolutely
necessary". But in that logic, the shell functions are not "absolutely
necessary", and even the command substitutions are not "absolutely
necessary" as everything can be in principle processed in combinations
of pipelines and { list; }, etc.
But they benefit the user tremendously.
OK, I understand what you try to say, (though I regard "no one ever"
and "any sense" as exaggerations; at least the developer who
introduced the feature should have had the reasoning and also should
have used it.
The developer doesn't count as an organic user. And ksh93 has been
around for three decades; if there were demand for the feature in
question, other shells would have copied it already.
I still think it benefits the users, even though it can be thought of
kind of syntax sugar for `list > tmp; var=$(< tmp)' (as managing the
temporary files properly is usually.non-trivial).
How many forks does that avoid anyway? Let's be realistic, the overhead
from one fork is not enough to warrant new syntax.
