Re: posix command search and execution

[Chet Ramey]

On 11/7/23 5:04 PM, Mike Jonkmans wrote:


   | > 1d) Lists 20 fixed utility names (like alias, cd etc.) that are
   | >  to be invoked at his point. No PATH search yet.
   | >   These are the `regular builtins'.
In the next standard the ones listed are the intrinsic builtins,
and includes only those that must be builtin to work.   But
implementations can add more to the list.


Chet mentioned that. But I find the Austin-discussion hard to read.
It makes sense to partition the builtins in three categories with
a separate name for each.


It depends on your requirements. If you want something that is easy to
explain to users, you want to reduce the distinction between `intrinsics'
and `regular builtins' to something as small as possible. That way the
search reduces to

special builtins*
functions
intrinsics
executables

in that order, with ways to override each one (except special builtins
in posix mode).




   | > 1eI) Search is successful.
   | > 1eIa) Check for `regular builtins' and functions
   | >   and invoke that regular builtin/function.
   | >   Q: Shouldn't this specify an ordering for builtins/functions?
   | The text seems to imply that you can't have both, doesn't it?
While I suppose you could have both, it would be very unusual.


Unusuality sketch:
- the shell provides a builtin for a standard utility
- the distributor provides a function for the same utility in /etc/profile
   (maybe to mitigate some security issue)

Are scripts in /etc/profile considered part of the implementation?


Yes. But there's no way to make a distinction between this type of function
and any other user-supplied function, so the distinction isn't useful.



   | Note that this seems to require that you can only run
   | a builtin if it exists (or something with that name exists) in $PATH.
A builtin for a standard utility, yes. 


The standard doesn't explicitly make that distinction.



I already can't find the definition of standard utilities.


The standard utilities are the ones specified in the standard.




   | I think the resolution to interpretation 854 addresses this. Shells
   | who want this ordering just declare all the builtins they implement as
   | `intrinsic' so they're not subject to a PATH search.
Yes.


Can this intrinsic list be amended with any user loaded builtins?


You mean enable -f? It's up to the shell implementation. Bash treats
dynamically-loaded builtins the same as any other builtin.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: posix command search and execution

[Chet Ramey]

On 11/7/23 5:37 PM, Mike Jonkmans wrote:

On Tue, Nov 07, 2023 at 11:49:25AM -0500, Chet Ramey wrote:

On 11/7/23 8:54 AM, Mike Jonkmans wrote:


...


Look at https://www.austingroupbugs.net/view.php?id=854 for a discussion
of this issue.

Thanks for the link, I find that very hard to read though.

It's also incomplete; there was a lot of discussion on the mailing list.
I don't have a link to a usable public mailing list archive.


So the discussion is hidden. Hmm.


Not hidden; if you look at the open group's mailing list archives, you
can find the messages. I just don't consider the interface usable. It's
barely searchable.


Then again, is there a requirement for the standard utilities to be
found in the current PATH? Or do they just need to be present somewhere.

They have to be findable using the value returned by `getconf PATH'. If
the user modifies PATH to, say, prepend directories before that standard
PATH, then all bets are off.


I see. Weirdly on Ubuntu 22.04, with /bin symlinked to /usr/bin,
`getconf PATH' produces `/bin:/usr/bin'.
That looks like a recipe for redundant `stats'.


Does that matter? The value getconf returns is static, and is guaranteed
to find all the standard utilities regardless of what the file system
looks like.


(But wait! The list of intrinsics in the latest draft includes type and
ulimit, and isn't XSI-shaded. So that will change.)


Isn't that described in `Note 0004803' in
https://www.austingroupbugs.net/view.php?id=854
(not possible to add shade to type and ulimit in column order)


Yes, good catch.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/

Re: posix command search and execution

[Robert Elz]

Date:Tue, 7 Nov 2023 23:04:10 +0100
From:Mike Jonkmans 
Message-ID:  <20231107220410.gc27...@jonkmans.nl>


  | In particular, a user function with the name of a standard utility,
  | will be called at this point.

Yes.   That much is very clear.

  | Chet mentioned that. But I find the Austin-discussion hard to read.

It can be, yes.  Particularly when you have to cope with stuff
being discussed in places that aren't public, but affects the results.

  | It makes sense to partition the builtins in three categories with
  | a separate name for each.

That's more or less what has been done now, with the specials, intrinsics,
and regular builtins.

  | Unusuality sketch:
  | - the shell provides a builtin for a standard utility
  | - the distributor provides a function for the same utility

What counts as "the implementation" is always (apparently) an unspecified
issue, splitting things into 2 camps (as almost every standards org
tends to do, there's "us" and "the rest of them") never works well
in practice, yet continues to be done for all kinds of purposes, as it
always seems to provide a nice clean answer - yet as that shows, never
really does, as what is called what always depends upon from what point
of view you're looking, and there tend to be many.

But:
  | in /etc/profile

in that case it would be not be "provided by the implementation" - anything
loaded from profile files is a user add on (even if it is forced upon the
user by their admins) - if the local user doesn't want that function, they
can simply remove it from /etc/profile (or not use that at all).

  | Are scripts in /etc/profile considered part of the implementation?

So, no, I don't think so.

  | It is all kind of theoretical. What wonders me is that the POSIX
  | specifications and definitions sometimes are imprecise or lacking.

They sometimes are, often because no-one considered some weird case,
so didn't think to actually write down what happens then - in the POSIX
world that's called implicitly unspecified.  Occasionally there are
simple errors (minor ones, like spelling, grammar, punctuation or
formatting issues - which are easy to overlook, though annoying) to
major ones where the standard says what someone thought was correct,
but they were mistaken.

But it isn't always the case, even when it seems so - to really
understand it, you really have to read and comprehend (almost) the
whole damn thing (all 4000 approx pages of it).   The text in one
place often says nothing about something which is specified in some
entirely different context.

  | > Yes.   But almost all shells implement it that way, so the
  | > seemingly logical assumption is mostly backed by experience.
  |
  | Again this is not too precise for a standard.

Yes, but the point is that this isn't what the standard says, more like
what it really should say.   This PATH search for builtins stuff is an
invention by people trying to force something upon shells that none of
them implemented, because they thought it would be better.

It is actually messier that it first seems, if you have a builtin command,
it isn't simply finding that command's name in a PATH search that is
required, but that the directory (from PATH) in which it was found be the
one "associated" with the builtin command (how exactly that is determined,
or what it really even means is not specified anywhere).   The effect is
that if a shell with a builtin "test" believes that the associated
directory is /bin and someone's path is /usr/bin:/bin - and there is a
test command in /usr/bin (even if /usr/bin and /bin are the same directory,
or /bin/name and /usr/bin/name are linked and so invoke the same thing)
then the filesystem command is supposed to be invoked rather than the
builtin.There are reasons for that, but they're really fairly stupid.

[Finding the standard utilities]
  | Good to know. I was wondering about that.
  | But where is that being specified?

Somewhere in the conformance stuff I believe it is, I don't have
the time to go look for it at the minute.

  | I already can't find the definition of standard utilities.

There really isn't one - it is just a shorthand for "utilities
specified by the standard" - less to type (and read).

  | Ok. I was not aware that all standard utilities have to be in PATH.

Not the utilities, or not the way you phrased that, the directories in
which the standard utilities are found have to be in PATH (and PATH needs
to be ordered such that they are found, not some other utility of the
same name).But I expect that's what you meant.

  | One might want to override `.' (source in bash parlance) for logging.

In a portable script you cannot, as POSIX only requires that "name"s work
as function names, and "." is not a "name" (a "name" is the same syntax as
used for variable names - which doesn't include special or positional params).

Most shells will allow it (POSIX doesn't forbid it) but you cannot rely
upon it working.

  | Can

Re: posix command search and execution

[Mike Jonkmans]

On Wed, Nov 08, 2023 at 10:38:19AM -0500, Chet Ramey wrote:
> On 11/7/23 5:04 PM, Mike Jonkmans wrote:
> 
> It depends on your requirements. If you want something that is easy to
> explain to users, you want to reduce the distinction between `intrinsics'
> and `regular builtins' to something as small as possible. That way the
> search reduces to
> 
> special builtins*
> functions
> intrinsics
> executables
> 
> in that order, with ways to override each one (except special builtins
> in posix mode).

That is a good point.
The POSIX standard is intended for two classes of readers, implementors and
users (application writers).
If the users had anything to say, they would probably vote for the bash way.

> > >| > 1eI) Search is successful.
> > >| > 1eIa) Check for `regular builtins' and functions
> > >| >   and invoke that regular builtin/function.
> > >| >   Q: Shouldn't this specify an ordering for builtins/functions?
> > >| The text seems to imply that you can't have both, doesn't it?
> > > While I suppose you could have both, it would be very unusual.
> > Unusuality sketch:
> > - the shell provides a builtin for a standard utility
> > - the distributor provides a function for the same utility in /etc/profile
> >(maybe to mitigate some security issue)
> > Are scripts in /etc/profile considered part of the implementation?
> Yes. But there's no way to make a distinction between this type of function
> and any other user-supplied function, so the distinction isn't useful.

A shell could flag a function as belonging to the implementation.
Then mark all new functions as such while not done with /etc/profile.
Not that I really would like to see that happen, just for argument's sake.

> > I already can't find the definition of standard utilities.
> The standard utilities are the ones specified in the standard.

An implementor will probably know this. But a user might not.
The standard should define that.

> > >| I think the resolution to interpretation 854 addresses this. Shells
> > >| who want this ordering just declare all the builtins they implement 
> > > as
> > >| `intrinsic' so they're not subject to a PATH search.
> > > Yes.
> > Can this intrinsic list be amended with any user loaded builtins?
> You mean enable -f? It's up to the shell implementation. Bash treats
> dynamically-loaded builtins the same as any other builtin.

That was indeed what I meant.
Does the new POSIX version with intrinsics also allow for this?


-- 
Regards, Mike Jonkmans

Re: posix command search and execution

[Mike Jonkmans]

On Wed, Nov 08, 2023 at 11:52:19PM +0700, Robert Elz wrote:
> Date:Tue, 7 Nov 2023 23:04:10 +0100
> From:Mike Jonkmans 
> Message-ID:  <20231107220410.gc27...@jonkmans.nl>
> 
>   | It makes sense to partition the builtins in three categories with
>   | a separate name for each.
> 
> That's more or less what has been done now, with the specials, intrinsics,
> and regular builtins.

A step in the right direction.
In 30 years or so we may lose the special builtins ;)

>   | Unusuality sketch:
>   | - the shell provides a builtin for a standard utility
>   | - the distributor provides a function for the same utility
> 
> What counts as "the implementation" is always (apparently) an unspecified
> issue, splitting things into 2 camps (as almost every standards org
> tends to do, there's "us" and "the rest of them") never works well
> in practice, yet continues to be done for all kinds of purposes, as it
> always seems to provide a nice clean answer - yet as that shows, never
> really does, as what is called what always depends upon from what point
> of view you're looking, and there tend to be many.
> 
> But:
>   | in /etc/profile
> in that case it would be not be "provided by the implementation" - anything
> loaded from profile files is a user add on (even if it is forced upon the
> user by their admins) - if the local user doesn't want that function, they
> can simply remove it from /etc/profile (or not use that at all).
> 
>   | Are scripts in /etc/profile considered part of the implementation?
> So, no, I don't think so.

I don't want to start a flame war, but Chet thought it was. :)

>From a user/application writer's perspective I would say that it is.
Having a work around is not really convincing, it is still part of the
implementation.

Code -> Distribution -> Admin -> User

The `implementation' could be including any up to the first three.

>   | It is all kind of theoretical. What wonders me is that the POSIX
>   | specifications and definitions sometimes are imprecise or lacking.
> 
> They sometimes are, often because no-one considered some weird case,
> so didn't think to actually write down what happens then - in the POSIX
> world that's called implicitly unspecified.  Occasionally there are
> simple errors (minor ones, like spelling, grammar, punctuation or
> formatting issues - which are easy to overlook, though annoying) to
> major ones where the standard says what someone thought was correct,
> but they were mistaken.
> 
> But it isn't always the case, even when it seems so - to really
> understand it, you really have to read and comprehend (almost) the
> whole damn thing (all 4000 approx pages of it).   The text in one
> place often says nothing about something which is specified in some
> entirely different context.

Writing specifications is not an easy task.
Let's try to make things better (for whoever is defining `better').

Seems there are at least a couple of goals for the POSIX standard.
- providing direction to implementers
- converging implementations
- providing an api to users

An anti goal would be to keep supporting historical burdens,
like special builtins.

>   | > Yes.   But almost all shells implement it that way, so the
>   | > seemingly logical assumption is mostly backed by experience.
>   | Again this is not too precise for a standard.
> Yes, but the point is that this isn't what the standard says, more like
> what it really should say.   This PATH search for builtins stuff is an
> invention by people trying to force something upon shells that none of
> them implemented, because they thought it would be better.

Ouch.

> It is actually messier that it first seems, if you have a builtin command,
> it isn't simply finding that command's name in a PATH search that is
> required, but that the directory (from PATH) in which it was found be the
> one "associated" with the builtin command (how exactly that is determined,
> or what it really even means is not specified anywhere).   The effect is
> that if a shell with a builtin "test" believes that the associated
> directory is /bin and someone's path is /usr/bin:/bin - and there is a
> test command in /usr/bin (even if /usr/bin and /bin are the same directory,
> or /bin/name and /usr/bin/name are linked and so invoke the same thing)
> then the filesystem command is supposed to be invoked rather than the
> builtin.There are reasons for that, but they're really fairly stupid.

Oh dear. I thought it could not get any worse.

> [Finding the standard utilities]
>   | Good to know. I was wondering about that.
>   | But where is that being specified?
> Somewhere in the conformance stuff I believe it is, I don't have
> the time to go look for it at the minute.
> 
>   | I already can't find the definition of standard utilities.
> There really isn't one - it is just a shorthand for "utilities
> specified by the standard" - less to type (and read).

It sounds logical, but it might be unwise to leave unspecified

Idea: jobs(1) -i to print only :%ID:s

[Steffen Nurpmeso]

Hello.

Well first i was thinking of a "%*" special so i can say
"kill %*", but get_job_spec() and usage does not look promising.

The task: close all jobs at once (and dreaming of
%ID1-%ID2,ID3-ID4, etc).  Ie i often (really) am doing things that
require many instances of less(1), or git(1) log, and such, but
then the task is done and all of those can (and shall) vanish at
once.  I mean, ok, i *could* create a recursive bash(1) and then
simply quit the shell, maybe, ... but often i do not know in
advance such a sprint starts.

The "problem" with the current way bash is doing it is that bash's
job handling does not recognize jobs die under the hood:

  $ jobs
  [1]-  Stopped LESS= less -RIFe README
  [2]+  Stopped LESS= less -RIFe TODO
  $ kill $(jobs -p)
  $

^ nothing

  $ jobs
  [1]-  Stopped LESS= less -RIFe README
  [2]+  Stopped LESS= less -RIFe TODO
  $ fg
  LESS= less -RIFe TODO

?=15

  $ fg
  LESS= less -RIFe README

?=15
  $

Compared to proper job stuff:

  $ jobs
  [1]   Stopped git loca
  [2]-  Stopped LESS= less -RIFe TODO
  [3]+  Stopped LESS= less -RIFe README
  $ kill %2 %3
  $
  [2]   Exit 15 LESS= less -RIFe TODO
  [3]-  Exit 15 LESS= less -RIFe README
  $

If there would be a "jobs -i" to properly unfold living jobs IDs
in equal spirit to -p that prints job's process IDs, then this
would be great.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: Defaults when cross-compiling

[Michael T. Kloos]

It seems to me that Autoconf (configure) is making some bad choices if 
it is just guessing that support exists like that, especially when it 
has a guaranteed fallback.  It's job is to setup the build for the 
target host system.  I was able to fix the build by using 
--without-bash-malloc.  Simply unsetting HAVE_SBRK didn't work by 
itself.  It is my opinion that the behavior should be that if configure 
can't check that support exists, it should change the default build 
options to be --without-bash-malloc.  If the user, then, re-enables it 
on the configure command line with --with-bash-malloc, that is on them.


On 11/7/2023 11:11 AM, Chet Ramey wrote:

On 11/7/23 10:03 AM, Oğuz wrote:
On Tuesday, November 7, 2023, Chet Ramey > wrote:


    It's interesting that musl supports brk but not sbrk


It doesn't support locales either. I always assumed it's someone's 
toy project but looks like there are Linux distros shipping it 
instead of glibc. Huh


They probably want some minimal system for containers. dash doesn't 
handle
multibyte characters or different locales either, but distros still 
use it.