Date: Thu, 23 Jul 2026 19:30:32 +0200
From: "[email protected]" <[email protected]>
Message-ID:
<ambp191mb2886d35b9cb62e932007fbffce...@ambp191mb2886.eurp191.prod.outlook.com>
| thank you for actually reading the patch and the tables -- this is exactly
| the review I was hoping for.
No problem.
| For scripts meant to travel, agreed without reservation -- which is why
| the feature is off by default,
That actually makes it worse. If it was on all the time, then at least
scripts would be portable to anything using a new enough bash (which is not
an unusual demand for many bash scripts, even when there's no compelling
reason to use bash only features). Having it optional (and probably disabled
at sites where few users, if any, would desire it) is worse than simply
having it on all the time.
I don't know how bash handles internal data when operating in a non
ascii, non UTF-8 locale, but if it used wchar_t for everything, and
converts, or encodes everything in UTF-8, so byte operations for most
things just work, you shouldn't need to even limit usage to when the
user is using UTF-8.
| But bash already lets people write deliberately non-portable
| scripts on an opt-in basis
I know, it is sad. [Insert crying emoji here.] Some of that is relatively
harmless, and can be worked around (use it if supported, use external
commands, functions, or whatever otherwise) but some of it affects the
syntax, such that simply having it present in the script means other shells
can't parse it, even if there would never be any attempt to execute it.
| Deliberate: UTF-8 is self-synchronizing and no byte of a multibyte
| sequence collides with ASCII syntax characters,
You don't need to sell me on the benefits of UTF-8.
| If you'd prefer [...]
Let me be quite explicit here, what I prefer is 100% irrelevant,
I have nothing whatever to do with bash development (I occasionally
make suggestions, or comments, but they are no more than that.)
| a bash built with an older table accepts a subset, everything it
| accepts stays valid under every future table,
Firstly, that assumes that they keep the promises that they have made,
and no-matter who it is who made that guarantee, or how sincere they
are, there is literally no way that anyone can guarantee anything about
the future.
Just assume for a minute that some insane US president decides that
some character currently classed as punctuation (say the US dollar
sign) should be a treated as a letter, has a friendly congress pass
legislation to that effect, and then slaps tarrifs, and or import
bans, on anything which doesn't implement the $ to be a letter.
What do you expect would happen then? (And yes, I know, '$' is
a terrible example for this.)
But ignoring that, if you need a new bash to get a new table of
characters, then you're just reducing portability even more - it would
be much better if the tables were loaded from an external file at
startup time, assuming they are going to get included, so any user
could update them any time they have the requirement, without needing
to undergo the risks that installing a new version of anything always
entails.
And while on the generic question of the tables - in the case that this
were every to be included, I have 2 more suggestions. First, you don't
really need 3 tables, that just adds complexity. Encode everything in a
single table - I have no idea what the "ccc" values mean (and while I'm
sure you could explain them to me, please don't bother), but it is currently
just a char field, so there will be padding added, so you could easily use
currently unused space that is in the MBID_CCC_RANGE type currently, and
encode the "start" "mid" (and the ccc) values in there, you'd only ever
need a single lookup/char. The table might end up using slightly more
space than the 3 current ones combined, but the simplicity of the result
more than makes up for that (or would).
And second, since it is clear that these tables (or table in the future
perhaps) is machine generated, that generation should be done as a part
of building bash, rather than simply dumping a whole bunch of hex values
with no explanations of what is what, or why, in the source.
| The proposal that really does decide forever is the other one: freezing
| "everything from 0x80 up, assigned or not" means today's unassigned code
| point can become tomorrow's combining mark or format control, silently
| changing how existing names normalize and display.
Yes, they can. Just as in file names, or anything else.
| Let me make that concrete, because no word-plausibility test exists
| anywhere in the patch -- I agree completely that names need not be words.
Yes, obviously.
| In U+0E00..U+0E7F the tables contain: all 55 Thai letters (category Lo)
| plus MAIYAMOK as start characters; SARA AM, all 16 combining vowel/tone
| marks (Mn) and the ten Thai digits as continue characters (they cannot
| lead a name, exactly like ASCII digits);
I will come back to the digits a bit later, but why (to start with) is
SARA AM excluded from the start characters? It is even an Lo category
character. (For those unfamiliar with Thai, "SARA" is just "vowel",
more or less, certainly close enough for us here, and "SARA AM" is much
like the vowel sound generated by the "um" dipthong, which is really
what it is in this case, in something like "hum" or "rum".)
Any why not the Mn category vowells? Is there some reason a Thai vowel
(or most of them) is not to be eligible to be a start character of a
variable name, where Latin vowels are? Why? Makes no sense to me
(unless you're concerned with what it would look like to someone who
understands how Thai is supposed to be used). I can (perhaps) understand
excluding the tone marks, those are more like diaresis or accents, etc,
in Latin, and one of those, standing alone, would be kind of peculiar
(though I'm not sure I'd actually exclude those either, Thai or Latin,
or anything else).
I can certainly see no reason to exclude the chess pieces, card suits,
and similar (there's a lot of stuff in unicode) from being var names
(by themselves, or in combination with other characters.
Further, and I certainly admit to knowing nothing whatever about anything
in this area, it looks to me as if the vast majority of the CJK codepoints
(other than the simplified alphabets) are encoded in unicode as So category.
Are you really going to also exclude all of those from being var names??
| That is the same letters/marks/digits vs. punctuation/symbols
| line that [A-Za-z0-9_] draws for ASCII,
except it isn't, all the ASCII vowels are OK as start chars, a shell
variable can be called "i" (or "I") why can't its Thai equivalent
(SARA I) ?
You seem to be determined to be compatible with C, Rust (and more), but
why? Bash isn't even compatible with its close relatives (the other
Bourne shell lookalike shells, POSIX compat or not), being compatible
with C or Rust or whatever is pointless.
Bash already has extensions beyond what C allows - C function names need
to be identifiers too, no such requirement in bash (and a good thing too).
Then for the digits (in Thai, or any other language with its own set),
why are they being excluded. Are you planning on making
eval echo \$$'\u0E50'
have the same result as
eval echo \$0
(which it is, except using the Thai 0, which looks more like an ascii 'o'
than a 0). Or are you expecting
eval echo $'$(( \u0E51\u0E52\u0E53 + \u0E54\u0E55\u0E56 ))'
to work (it doesn't currently), and if that were to be made to work,
would the result be "\u0E55\u0E57\u0E59" or "579" ? How about
eval echo $'$(( \u0E51\u0E52\u0E53 + 456 ))'
Those don't work now, and I can't think of any reason they ever should.
Even in Thailand, the Thai digits aren't used for that kind of thing,
Arabic numerals are used essentially everywhere, except some ceremonial
purposes (currency carries both versions). Given that there is no reason
to treat the Thai digits as being actual digits (and doing so would create
all kinds of problems, such as what to do with that last example), why have
any restriction on where a Thai digit (or any other non-ascii digit)
is allowed to appear in an identifier.
The only reasons the limitations (on ascii digits) exist now are to support
those kinds of uses - and if $0 $1 (etc) in the original Bourne shell weren't
treated differently for parsing reasons, we probably wouldn't have that
restriction in any shell identifiers, even though that would make use of
var names in arithmetic without the '$' impossible -- arithmetic like
this didn't exist when the syntax rules were established).
| any combination of Thai letters is a valid name,
Except it isn't, you have excluded about half (I didn't count) the vowels
from being starter characters, and vowels are letters. It wouldn't be
possible (even if no Thai would ever want to do it) to write the Thai
equivalent of
i=1
because SARA I isn't allowed as a starter (and for those reading, if
you pronounce that "i" as "eye" then it would be different, as that is
a different vowel, the one in question sounds like the "i" in "hit" or
"miss", when you read "SARA I" you should say it that way, not "eye",
or it would be confused with "SARA AI MAIMUAN" or "SARA AI MAIMALAI",
which are "sara eye" - and yes there are two, so they are distinguished
by giving a word which uses each, the same way the consonants are, Thai
has a bazillion different 'K' characters, and more than one of most
consonants).
| If some specific code point in there struck you as wrong, please name it:
Done above, SARA I (\u0E34) is one, as are SARA II, SARA U, SARA UU,
and more, you can work out the others.
| the table is generated, so a
| discrepancy would be a generator bug and I very much want to know.
But it isn't that, it is a design decision to exclude Mn category chars.
(and for some reason SARA AM, \u0E33, which is Lo category). I didn't
check everything.
| � 1. Invisible characters. With ZWSP allowed, $Stra�e and
| � � �$Stra�e<U+200B> are two different variables that render
| � � �identically in every terminal and diff;
Terminal, yes, but if diff doesn't find the difference, even if
the result isn't obvious when displayed, then something is broken.
| � 2. Bidi format controls are also "not sh syntax", and allowing them
| � � �in names lets displayed order diverge from byte order in the
| � � �middle of a command line.
But they can occur in any other shell word. What's the difference if
they also occur in those words which are restricted to be names ?
Generic shell words (args to commands for example - even command names)
can be anything except the reserved operator, quoting, and white space
characters, and include any of those as well, if quoted.
| � 3. Aliasing without normalization: precomposed a-umlaut and
| � � �a + U+0308 would be two distinct, visually identical variables --
| � � �an equivalence-class problem ASCII never had. The patch's
| � � �NFC-or-reject rule keeps byte equality == identity, so grep and
| � � �humans stay reliable.
I have no idea what grep has to do with it, it should have no issues
detecting the differences, but yes, as with zero width chars, and the
spaces that aren't 0x20 or 0x08, that kind of thing can cause issues for
humans. I'm not sure it is enough of a reason to ban them however.
To code they are clearly different things, there's no ambiguity at all,
just human confusion.
| v2 (attached) applies exactly that profile
Just in case it matters, I used your v3 from the later message when
I was checking things for this message.
| Under anything-goes, yes -- under this patch no, and I'd argue that's
| the consistent side of the line: emoji are symbols (So), the same
| category as $ and ~, not letters.
Yes, they are, that I'd agree with. But aside from the sh syntax
chars, I see no reason why symbols shouldn't be able to appear in
variable names. Even now, there's no good reason I shouldn't be
able to do
Three*Six=18
is there?
And then reference it as ${Three*Six}, or even just $Three*Six
In sh (incl bash) '*' (and lots more like it) is just a word character,
nothing particularly special about it. The ascii ones wouldn't work
as starter chars, as most of them have been assigned as special parameter
names, or for other uses (like ${ etc) but there's no reason most of them
('}' is an obvious exception) couldn't appear in other places in var
names (the ones used as operators in variable expansions would need to
be omitted). But all these weird rules only apply to the ascii chars
that have other defined meanings in the syntax. All the values > 0x80
don't, and could easily be used anywhere.
I think you're excluding many of them, just to be compatible with other
languages. Shell needs almost none of that, it has a very simple syntax,
in general, and can cope with almost anything. Let it.
| So: I'll gladly split out the shopt return-value fix, add explicit
| guards if preferred, and rebase onto devel.
For any of that you'd need an opinion from Chet, not from me.
kre
