Félix, Léa, Wiley, thank you all for replying to me directly. I have since applied v5 and run the codebase I am migrating through it, so let me answer with measurements from that migration rather than with opinions. Wiley, your correction is the one I most needed, so it gets a section of its own further down.
For reproducibility: the artefact tested is v5 as posted to this list, taken from the author's repository at https://github.com/Otzie2023/Bash-with-Unicode-Identifier applied to the bash-5.3 release tarball. Anyone wanting to check the character-level claims below without building can read them straight out of the generated table in mbidtbl.h. Two things up front. Your examples do not measure the patch: Félix's first script uses associative array subscripts, which are arbitrary strings in bash and always have been -- an untouched bash 5.2.21 produces the same eighteen keys here. His second measures zsh, which I am now running. Neither shows what the patch does. And the patch, tested, does the thing I need and zsh does not. What the patch does with the names I have to migrate ---------------------------------------------------- The roughly 1,500 identifiers I am migrating draw on this character inventory, freely combined with ASCII letters and underscores: αβγδεζηθικλμνξοπρστυφχψω ΓΔΘΞΣΦΨΩ áéíóúñÑÁÉÍÓÚ Greek for physical quantities, Spanish for the internal glossary -- forty-four characters, all NFC-stable. Every one is accepted, and so is every name built from them: $ αβγδεζηθικλμνξοπρστυφχψωΓΔΘΞΣΦΨΩáéíóúñÑÁÉÍÓÚ=4 $ On stock bash that line is "command not found". This is the first time since the migration started that these names have worked in bash, and I wanted the list to have that data point, because so far the discussion has been about hypothetical identifiers and I appear to be the only person here with real ones. Félix, your six sigmas -- the answer, and why I want it ------------------------------------------------------- Léa suggested zsh as the natural path. I had already taken it, so treat this as a report from that path: zsh 5.9 on Ubuntu accepts all six of Félix's sigmas as distinct variables. I checked. Under the patch, one of his eighteen spellings is a legal name, U+03A3 -- the five mathematical sigmas are Identifier_Type=Not_NFKC and the security profile removes them, and the variation selectors went a revision earlier. That is the answer I want, and not because six sigmas are a plausible attack on this codebase. It is because these names are read by people, in review and in diffs, and I would rather be in the shell that says no to five of six than the one that says yes to all six and warns about nothing. Félix, I think your example argues for a defined repertoire, not against the feature. A warning about the next revision, from the affected side ---------------------------------------------------------- This is the part I most want on the record, because it concerns something that has been offered in this thread but not yet built. Mixed-script restriction under the UTS #39 restriction levels was proposed earlier as a possible addition, with the author saying he would implement whichever default the maintainer preferred. Please weigh this codebase before choosing rejection. The identifiers here mix Greek and Latin inside a single name -- that is what "Greek quantity plus ASCII qualifier" means in practice. Latin+Greek is explicitly excluded from the Moderately Restrictive level, so such names sit at Minimally Restrictive and would be rejected under any Highly Restrictive rule. Chunking the name at case and punctuation boundaries first, as UTS #55 section 5.1.2 does, saves the ones written Σ_total or θ_max, because the underscore splits them into single-script chunks. It does not save the ones written without a separator, and there are many of those. So: as a warning, useful. As a rejection, it would undo exactly what the patch just fixed here. I should be straight about the other side of this, since it is the strongest thing anyone could say against my position. Eleven of the thirty-two Greek letters in use here have Latin skeletons under UTS #39 -- α→a, ο→o, ν→v, ρ→p, ι→i, γ→y, υ→u, σ→o among them. That means this codebase genuinely contains names that can be confused with Latin ones. I am not going to pretend otherwise. But that exposure exists today in zsh, where nothing is defined at all, and it exists today in bash function names, which take any bytes at all. It is a review and tooling problem, and I would rather have it in a shell that has a specified repertoire and a normalization rule than in one that has neither. Wiley: you are right and my sentence was wrong ----------------------------------------------- "A shell that aims to ensure portability should allow more characters rather than fewer" is not defensible as written, and your definition is the correct one. Portability in shell scripting is the common subset; fewer features used means more shells can run the script. Allowing more characters does not produce that, and I should not have used the word. What I was actually describing is a different problem that I gave the wrong name: moving an existing body of code between platforms. That is not about the common subset at all, because these scripts were never in it and never will be -- they came from PowerShell and zsh and they use arrays, [[ and much else. I was never going to run them on dash or on an EOL'd HP-UX box. The question was only whether bash could be a target at all, and today it cannot be. Two different things, one word, and I picked the wrong one. On "bashism": yes, exactly, and I do not think anyone has claimed otherwise -- the patch is off by default, gated behind a file only root can create, and its own manual text warns that such scripts are not portable. It is a bashism in the same sense as arrays, [[, local, process substitution and extglob. Naming the category is not yet an argument against it; the argument would have to be about cost. With the switch off the behaviour is byte-for-byte that of an unpatched bash, so the cost to anyone not using it is zero. One thing I did test, because your framework is the right one to test it in. Bashisms differ a great deal in how they fail elsewhere, and this one is in the well-behaved class: $ cat t.sh Σ=1 echo "status: $?" echo "expansion: [$Σ]" $ dash t.sh t.sh: 1: Σ=1: not found status: 127 expansion: [$Σ] $ mksh t.sh E: t.sh[1]: Σ=1: inaccessible or not found status: 127 $ busybox ash t.sh t.sh: line 1: Σ=1: not found status: 127 The assignment errors out with a diagnostic and status 127 in every POSIX shell I tried, and the expansion degrades to the literal characters rather than to a plausible wrong value. Compare `echo -e`, which silently prints something different in dash than in bash. If one of these scripts ever escaped onto a machine without the feature, it would stop at the first line rather than run and produce quiet nonsense. That is not an argument for the feature, but it is worth knowing before deciding how dangerous this particular bashism is. Finally, a small note on your last sentence. It is already a zsh-ism, and was reported earlier in this thread to be a ksh93-ism and a fish-ism as well. Bash adopting it would make the set of shells that accept such names larger, not smaller. Léa: on the same word --------------------- The above applies to your reply too, so I will not repeat it. The one thing I would add is that the dependencies you list -- fonts, editors, input methods, terminals -- already bind this project, and bash already carries them: these characters are in the filenames, Email Addresses, usernames (not only GUI-names), commit messages, documentation and string literals, all of which bash handles today without complaint. Variable names were the only place that broke. Regards, Ruben Vanlus Disclosure, since it is the kind of thing that should be stated rather than discovered: the author of this patch works at the same company I do. He has nothing to do with the migration described above -- a different project, no overlap -- and I hit the bash limitation through that work, not through his. I mention it because my only value to this discussion is that I have real names to migrate, and you should be able to weigh that knowing the connection exists. translated with deepl.com (free version)
