Hi Paul,
> >> Although I'm sure mbiter can be improved I
> >> don't see how it could catch up to mbcel so long as it continues to
> >> solve a harder problem than mbcel solves.
> >
> > I don't know exactly what you mean by "harder problem".
>
> I meant that it solves a harder porting problem because it worries about
> more issues, e.g., it worries about mbrtoc32 returning (size_t) -3,
This makes only for a small performance difference. I could measure it
by doing the benchmarks of
mbiterf-bench-tests mbuiterf-bench-tests
versus
mbiterf-bench-tests mbuiterf-bench-tests mbrtoc32-regular
In the latter case, the lines marked with
#if !GNULIB_MBRTOC32_REGULAR
are optimized away.
> or returning (size_t) -1 in the C locale.
Indeed, this shows as a difference between mbiterf and mbcel in the
test cases c, f:
mbiterf mbcel mbuiterf
c 1.145 0.670 1.179
f 13.028 5.714 14.654
But since the glibc people are already working on resolving this issue,
I won't spend time optimizing it one way or the other.
> > The other significant difference that I see is the handling of multibyte
> > sequences. When there 2 or 3 bytes (of, say, UTF-8) that constitute an
> > incomplete multibyte character at the end of the string,
>
> This isn't a problem for programs like grep and diff, where there's
> always a newline at the end the input buffer.
I disagree: Any program can run into it when the input is
<some valid UTF-8 characters><an incomplete UTF-8 character><newline>
My screenshot from the 'src/diff -y -t' output in an xterm also shows
that there is an issue.
> > - mbcel returns each byte, one by one, as a character without a
> > char32_t code.
>
> (A nit: it's not a character; it's an encoding error.)
Sure. Some programs then treat that error as if an U+FFFD character
had been read.
> > - ISO 10646 says ([1] section R.7) "it shall interpret that malformed
> > sequence in the same way that it interprets a character that is outside
> > the adopted subset".
>
> If I understand this requirement correctly mbcel satisfies it, as mbcel
> treats those two things in the same way, namely, as sequences of
> encoding error bytes.
No, I don't think mbcel satisfies it, since mbcel interprets the
"malformed sequence" not like "a character" but like multiple characters.
> > - Markus Kuhn's example ([2] section 3) has a section where
> > "All bytes of an incomplete sequence should be signalled as a single
> > malformed sequence, i.e., you should see only a single replacement
> > character in each of the next 10 tests."
>
> Kuhn is talking about programs that display characters to users and that
> need some way to signal encoding errors. But diff is not such a program:
> it doesn't need to display a signal for an incomplete sequence, because
> it's not responsible for display.
Kuhn's writeup is generally about UTF-8 decoding. In the year 2000, when it
was written, the most important decoders were in display engines of terminal
emulators. Nowadays, we have UTF-8 decoders in many many programs.
The Unicode Standard has several pages about this topic:
Unicode 15.0 section 3.9
https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf pages 124..129
It is also referenced by section 5.22
https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf page 255.
The relevant text starts at page 127:
"U+FFFD Substitution of Maximal Subparts
An increasing number of implementations are adopting the handling of
ill-formed subsequences as specified in the W3C standard for encoding
to achieve consistent U+FFFD replacements. ...
Although the Unicode Standard does not require this practice for conformance
..."
See also the table 3-11 on page 128.
So, clearly, this is not a *requirement* for a conforming UTF-8 decoder.
But the Unicode Standard's authors would not describe it in this great length
if it wasn't a good practice.
> It's certainly not
> typical practice in the GNU/Linux world. It's not true of the first five
> applications that I tested on Ubuntu 23.04: Emacs, Chrome, Firefox,
> less, and gnome-terminal.
Well, then I'll have to write a couple of QoI (quality of implementation)
reports...
(xterm does it right, but you are right that nowadays gnome-terminal and other
vte-based terminal emulators are the majority.)
> Even if Kuhn's suggestion were good for display programs, programs like
> diff should not treat differing encoding error byte sequences as if they
> were equivalent. If two files A and B contain different encoding errors
> I expect most users would prefer "diff A B" to report the differences.
Sure. If you were to use the 'mbiterf' module instead of mbcel, the
mb_equal macro from mbchar.h does the right thing. Yes, an mb_equal
call is a bit more complicated than the same_ch_err definition that
you have in diffutils/src/io.c. That's the unavoidable consequence of
treating a sequence of 2 or 3 bytes as *one* error.
> There's not
> even a standard column width for U+FFFD itself: Kuhn recommends 1 in
> <https://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c>, but 2 is more common in
> my experience.
Indeed, width considerations of strings with control characters are
hairy. And U+FFFD counts as a control character, according to
wcwidth(0xFFFD) == -1
(on glibc systems).
> > This may be acceptable as a corner case for 'diff'. But for a module offered
> > by Gnulib, we should IMO continue to follow the best practice here.
>
> Although Kuhn's suggestion may be best practice for some applications,
> it's not best for applications like diff, and it would be helpful if
> Gnulib could support these applications.
According to what I read in the Unicode Standard (above), it's a best practice
for all kinds of applications.
I'm not asking to rewrite the new code that you added in 'diff'. But for
other programs, from 'ls' to 'sed', I continue to think it would be a good
idea to follow that best practice.
Bruno
