[Bug binutils/34902] New: strings -U: invalid UTF-8 (overlong, surrogates, > U+10FFFF, F8-FF leads) is accepted as characters, and -U locale prints only the first byte of each multibyte character

[martin.mirchev at u dot nus.edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=34902

            Bug ID: 34902
           Summary: strings -U: invalid UTF-8 (overlong, surrogates, >
                    U+10FFFF, F8-FF leads) is accepted as characters, and
                    -U locale prints only the first byte of each multibyte
                    character
           Product: binutils
           Version: 2.48 (HEAD)
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: binutils
          Assignee: unassigned at sourceware dot org
          Reporter: martin.mirchev at u dot nus.edu
  Target Milestone: ---

The --unicode documentation for strings says the option "Controls the display
of UTF-8 encoded multibyte characters": invalid treats them as non-graphic,
and locale/hex/escape/highlight treat them as string characters and display
them in the locale, as <hex>, or as \uxxxx. There are two problems.

1. Invalid UTF-8 is accepted as characters.

is_valid_utf8() in binutils/strings.c (and the same inline checks in
print_unicode_stream_body) only checks the continuation-byte bit patterns.
The length comes from bits 0x20/0x10 of the lead byte, so these are all
treated as valid:

  - overlong forms: C0 80, C1 BF, E0 80 AF, F0 80 80 AF
  - UTF-16 surrogates: ED A0 80 (U+D800), ED BF BF (U+DFFF)
  - code points above U+10FFFF: F4 90 80 80, F7 BF BF BF
  - lead bytes F8-FF, which are read as 4-byte sequences

These bytes become part of the reported string (hex shows them as one
character), and escape prints code points that cannot exist, e.g.
"abcd\u0000efgh" for C0 80, "\ud800" for a surrogate, "ᄀ00" and
"῿ff" above the Unicode range. In escape mode an overlong sequence
shows as the character it spells: E0 80 AF prints as / ("/"). nm.c,
objdump.c and readelf.c have checks with the same structure for their
--unicode options.

2. -U locale writes only the lead byte.

In display_utf8_char() the locale case is

    case unicode_locale:
      printf ("%.1s", buffer);

which writes one byte, while the function returns the full sequence length,
so the continuation bytes are skipped. So "naïve café €10" (valid UTF-8) is
printed as "na\xc3ve caf\xc3 \xe210". The output is never valid UTF-8 if the
input contains any multibyte character, in any locale (checked with LC_ALL=C
and C.UTF-8). nm's locale mode copies all the bytes. Ubuntu's 2.38 strings
gives the same output, so this has been the case since --unicode was added.

Tested on master 1ec1cee48577c7b1960ddce5a6054b3f9fab6ef5 (GNU strings
2.47.50.20261027), Ubuntu 22.04, gcc 11 -g -O2. There are no sanitizer
findings: ASan+UBSan is clean on all inputs.

Attached (all files in strings-U-utf8-validation.tar.gz): gen.py writes one
file per case ("abcd" + sequence + "efgh"),
all.bin (the cases separated by NULs) and utf8.s (the same bytes in .data,
for the section-buffer path). Also valid.txt and run.sh.
Each sequence was also checked with Python's bytes.decode("utf-8"); the
column after the name shows the result.

  case                         decode   -U e                -U x
  valid-2byte-U+00E9           valid    abcdéefgh      abcd<0xc3a9>efgh
  valid-4byte-U+1F600          valid    abcdǶ00efgh    abcd<0xf09f9880>efgh
  overlong-C0-80-U+0000        INVALID  abcd\u0000efgh      abcd<0xc080>efgh
  overlong-C1-BF-U+007F        INVALID  abcd\u007fefgh      abcd<0xc1bf>efgh
  overlong-E0-80-AF-slash      INVALID  abcd/efgh      abcd<0xe080af>efgh
  overlong-F0-80-80-AF         INVALID  abcd\u00002fefgh   
abcd<0xf08080af>efgh
  surrogate-ED-A0-80-D800      INVALID  abcd\ud800efgh      abcd<0xeda080>efgh
  surrogate-ED-BF-BF-DFFF      INVALID  abcd\udfffefgh      abcd<0xedbfbf>efgh
  above-F4-90-80-80-110000     INVALID  abcdᄀ00efgh    abcd<0xf4908080>efgh
  above-F7-BF-BF-BF            INVALID  abcd῿ffefgh    abcd<0xf7bfbfbf>efgh
  lead-F8-80-80-80             INVALID  abcd\u000000efgh   
abcd<0xf8808080>efgh
  lead-FF-BF-BF-BF             INVALID  abcd῿ffefgh    abcd<0xffbfbfbf>efgh

(strings -a -U e FILE / strings -a -U x FILE.) For the invalid rows one
would expect "abcd" and "efgh" as two separate strings, as -U i prints. The
section-buffer path (strings -d -U x utf8.o, via print_unicode_buffer) gives
the same twelve hex lines.

  $ od -An -tx1 valid.txt
   6e 61 c3 af 76 65 20 63 61 66 c3 a9 20 e2 82 ac 31 30 0a
  $ strings -a -U l valid.txt | od -An -tx1
   6e 61 c3 76 65 20 63 61 66 c3 20 e2 31 30 0a

A likely fix is to reject C0, C1 and F5-FF leads, overlong E0/F0 second
bytes, ED A0-BF and F4 90+ in one shared check, and to write all utf8_len
bytes in the locale case.

This issue was found by an LLM-Enhanced Static Analysis Engine and verified
manually.

-- 
You are receiving this mail because:
You are on the CC list for the bug.