Thomas,
On Jul 23 05:44, Thomas Wolff via Cygwin wrote:
> > > Am 22.07.2025 um 15:05 schrieb Corinna Vinschen:
> > > > mbrtowc() is inherently a bad idea when it comes to UTF-16. It's a
> > > > function which only works really correctly for the unicode base plane,
> > > > or if wchar_t is big enough.
> > > >
> > > > It's the reason we don't use mbrtowc() if possible. It's better
> > > > to call
> > > > mbstowcs() or friends and allow at least 3 chars in the wchar_t buffer.
> > > > You can't change that in mintty by any chance?
> > [...]
> OK, suppose I'd consider to switch to mbs[[n]r]towcs, collecting bytes until
> the function gives me a result.
> This would work fine as long as I receive only valid sequences. But look at
> input string test case
> char nonbmp[] = {0xF8, 0x88, 0x8A, 0xAF, 0x2D, 0}; // an invalid sequence
> followed by a valid char
> The functions only return -1 and (in the case of mbsnrtowcs) do not advance
> the input pointer.
> So how am I supposed to recognize that the invalid sequence has ended and a
> valid character has arrived?
Apart from that, you probably still have a problem in mintty: GB18030.
The problem with GB18030 is, that you need all four bytes to generate
the high surrogate.
Consider the following GB18030 string: 0x90 0x30 0x81 0x30
This string translates into a UTF-16 surrogate pair: 0xd800 0xdc00.
If you run a tweaked version of your test applicaton from
https://cygwin.com/pipermail/cygwin/2025-July/258513.html:
setlocale (LC_CTYPE, "zh_CN.gb18030");
mb (0x90);
mb (0x30);
mb (0x81);
mb (0x30);
Then the output is:
90 -> 0000 : -2
30 -> 0000 : -2
81 -> 0000 : -2
30 -> D800 : 0
However, if you notice this situation...
if (ret_from_mbrtowc == 0 && codeset == gb18030
&& (pwc & 0xfc00) == 0xd800)
...you can just add a fake NUL byte:
mbrtowc (&wc, '\0', 1, &mbstate);
If you do that, the above sequence becomes:
90 -> 0000 : -2
30 -> 0000 : -2
81 -> 0000 : -2
30 -> D800 : 0
00 -> DC00 : 1
I hope this helps, if you didn't already handle GB18030 differently
in mintty.
Corinna
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation: https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
