On Mon, 12 Aug 2019, Lewis Hyatt wrote:
> Hello-
>
> The attached patch for libcpp adds support for extended characters (e.g.
> UTF-8)
> in identifiers. A preliminary version of the patch was posted on PR c/67224 as
> Comment 26 (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67224#c26) and
> discussed with Joseph Myers. Here is an updated patch incorporating all
> feedback received so far. I hope it is suitable now; please let me know if I
> can do anything else to make it ready for you to apply. I am happy to work on
> it further, whatever is needed. I can't easily test on anything other than
> x86_64-linux though. I did bootstrap all languages and run all tests on that
> platform, everything was good.
>
> The (relatively short) changes to libcpp are included inline here. I attached
> the test cases as a gzipped patch to avoid any problems with the encoding (the
> test cases contain some invalid UTF-8 and also other encodings such as latin-1
> as part of the testing).
>
> Thanks for taking a look at it!
Thanks, I think this is OK with a few updates to the documentation.
Specifically:
cpp.texi says:
In the 1999 C standard, identifiers may contain letters which are not
part of the ``basic source character set'', at the implementation's
discretion (such as accented Latin letters, Greek letters, or Chinese
ideograms). This may be done with an extended character set, or the
@samp{\u} and @samp{\U} escape sequences. GCC only accepts such
characters in the @samp{\u} and @samp{\U} forms.
and it's no longer accurate to say that only the \u and \U forms are
accepted.
cpp.texi, section "Implementation-defined behavior", discusses
implementation-defined characters in identifiers. It should say that GCC
accepts exactly those multibyte characters that correspond to UCNs for
characters permitted by the chosen version of the C or C++ standard.
cppopts.texi documents -fextended-identifiers as "Accept universal
character names in identifiers.". That needs to say the characters are
also accepted directly in the identifiers.
I should also note that a few of the tests added by the test are testing
things that are properties of the implementation that might arguably be
bugs, rather than standard features, and so perhaps should at least have
comments added saying they are testing those implementation properties.
gcc/testsuite/gcc.dg/cpp/ucnid-7-utf8.c, testing invalid UTF-8, is relying
on GCC, in its default -finput-charset=utf-8 mode, not actually checking
that the input is valid UTF-8. It's clear that avoiding such a check
makes sense in strings and comments, both as a matter of efficiency and
because it's likely to do the right thing for a lot of user programs that
use non-UTF-8 character sets in those places and just need the bytes in
the strings to be passed through to the compiler output (rather than
requiring users to specify -finput-charset and -fexec-charset for those
programs). Outside those contexts it's less obvious what's the best way
to behave (this sort of test, where the stray non-UTF-8 bytes are in text
that disappears as a result of macro expansion, is certainly a corner
case).
gcc/testsuite/g++.dg/cpp/ucnid-2-utf8.C and
gcc/testsuite/g++.dg/cpp/ucnid-3-utf8.C are testing double stringizing in
C++, where strictly the results they expect show that GCC does not conform
to the C++ standard requirement to convert all extended characters to UCNs
(because C++ does not have the special C rule making it
implementation-defined whether the \ of a UCN in a string literal is
doubled when stringizing).
--
Joseph S. Myers
jos...@codesourcery.com
