I spent some more time trying to understand the beast that's mbrtowc(3). uwe helped me by discussing over beer and we compared the C99 standard with the current man page. This provided quite a bit of additional insight which prompted me to more or less rewrite the entire page. The resulting diff is below. Please try to find errors!
I'm not very happy anymore with the code example I posted in http://marc.info/?l=openbsd-tech&m=129029477620897&w=2 but will update that in a separate diff. I've also renamed the variable names so chances are tedu will like this better :) Index: mbrtowc.3 =================================================================== RCS file: /cvs/src/lib/libc/locale/mbrtowc.3,v retrieving revision 1.2 diff -u -p -r1.2 mbrtowc.3 --- mbrtowc.3 31 May 2007 19:19:29 -0000 1.2 +++ mbrtowc.3 3 Dec 2010 18:14:10 -0000 @@ -28,145 +28,198 @@ .Dd $Mdocdate: May 31 2007 $ .Dt MBRTOWC 3 .Os -.\" ---------------------------------------------------------------------- .Sh NAME .Nm mbrtowc .Nd converts a multibyte character to a wide character (restartable) -.\" ---------------------------------------------------------------------- .Sh SYNOPSIS .Fd #include <wchar.h> .Ft size_t -.Fn mbrtowc "wchar_t * restrict pwc" "const char * restrict s" "size_t n" \ -"mbstate_t * restrict ps" -.\" ---------------------------------------------------------------------- +.Fn mbrtowc "wchar_t * restrict wc" "const char * restrict s" "size_t n" \ +"mbstate_t * restrict mbs" .Sh DESCRIPTION The .Fn mbrtowc -usually converts the multibyte character pointed to by -.Fa s -to a wide character, and stores the wide character +examines at most +.Fa n +bytes of the multibyte character byte string pointed to by +.Fa s , +converts those bytes to a wide character, and stores the wide character in the wchar_t object pointed to by -.Fa pwc +.Fa wc if -.Fa pwc -is non-null and +.Fa wc +is not null and .Fa s points to a valid character. -The conversion happens in accordance with the conversion state -described in the mbstate_t object pointed to by -.Fa ps . -This function may examine at most -.Fa n -bytes of the array beginning from +.Pp +The conversion will use at most +.Dv MB_CUR_MAX +bytes of the byte string pointed to by .Fa s . +.Dv MB_CUR_MAX is always smaller or equal to +.Dv MB_LEN_MAX . .Pp -If -.Fa s -points to a valid character and the character corresponds to a null wide -character, then the +Conversion happens in accordance with the conversion state described +by the mbstate_t object pointed to by +.Fa mbs . +The mbstate_t object pointed to by +.Fa mbs +must be initialized to zero before the application's first call to +.Fn mbrtowc . +.Fa mbs +can safely be reused without reinitialization after successful conversion. +.Pp +The behaviour of +.Fn mbrtowc +is affected by the +.Dv LC_CTYPE +category of the current locale. +If the locale is changed without reinitialization of +.Fa mbs , +the behaviour of .Fn mbrtowc -places the mbstate_t object pointed to by -.Fa ps -to an initial conversion state. +is undefined. .Pp Unlike .Xr mbtowc 3 , -the .Fn mbrtowc -may accept the byte sequence pointed to by +will accept an incomplete byte sequence pointed to by .Fa s -not forming a complete multibyte character -but which may be part of a valid character. -In this case, this function will accept all such bytes -and save them into the conversion state object pointed to by -.Fa ps . -They will be used at subsequent calls of this function to restart -the conversion suspended. +which does not form a complete character but is potentially part of +a valid character. +In this case, +.Fn mbrtowc +will save all such bytes into the conversion +state object pointed to by +.Fa mbs . +They will be used during subsequent calls of +.Fn mbrtowc +to restart the conversion suspended. .Pp -The behaviour of the +In state-dependent encodings, +.Fa s +may point to a special sequence of bytes called a +.Dq shift sequence . +Shift sequences switch between character encodings available within an +encoding scheme, e.g. between one-byte characters and two-byte characters. +One encoding scheme using shift sequences is Extended UNIX Code (EUC). +Shift sequence bytes correspond to no individual wide character, so .Fn mbrtowc -is affected by the -.Dv LC_CTYPE -category of the current locale. +treats them as if they were part of the subsequent multibyte character. .Pp -These are the special cases: +Special cases in interpretation of arguments are as follows: .Bl -tag -width 012345678901 -.It "s == NULL" -.Fn mbrtowc -sets the conversion state object pointed to by -.Fa ps -to an initial state and always returns 0. -Unlike -.Xr mbtowc 3 , -the value returned does not indicate whether the current encoding of -the locale is state-dependent. +.It "wc == NULL" +The conversion from a multibyte character to a wide character is performed +and the conversion state may be affected, but the resulting wide character +is discarded. .Pp -In this case, +This can be used to find out how many bytes are contained in the +multibyte character pointed to by +.Fa s , +which is a number between 1 and +.Dv MB_CUR_MAX +upon successful conversion. +.It "s == NULL" .Fn mbrtowc ignores -.Fa pwc +.Fa wc and .Fa n , -and is equivalent to the following call: +and behaves equivalent to .Bd -literal -offset indent -mbrtowc(NULL, "", 1, ps); +mbrtowc(NULL, "", 1, mbs); .Ed -.It "pwc == NULL" -The conversion from a multibyte character to a wide character has -taken place and the conversion state may be affected, but the resultant -wide character is discarded. -.It "ps == NULL" +.Pp +which attempts to use the state object pointed to by +.Fa mbs +to start or continue conversion using the empty zero-terminated string +as input, and discards the conversion result. +.Pp +If conversion succeeds, this call always returns zero. +Unlike +.Xr mbtowc 3 , +the value returned does not indicate whether the current encoding of +the locale is state-dependent, i.e. uses shift sequences. +.It "mbs == NULL" .Fn mbrtowc uses its own internal state object to keep the conversion state, instead of -.Fa ps -mentioned in this manual page. +.Fa mbs . +This internal conversion state is initialized once at program startup, +and is undefined after an encoding error occurred. +It is not safe to call +.Fn mbrtowc +again with a NULL +.Fa mbs +argument if +.Fn mbrtowc +returned (size_t)-1. .Pp Calling any other functions in .Em libc -never change the internal -state of -.Fn mbrtowc , -which is initialized at startup time of the program. +never changes the internal +conversion state object of +.Fn mbrtowc . .El -.\" ---------------------------------------------------------------------- .Sh RETURN VALUES -In the usual cases, -.Fn mbrtowc -returns: .Bl -tag -width 012345678901 .It 0 -The next bytes pointed to by +The bytes pointed to by .Fa s form a null character. .It positive -If .Fa s -points to a valid character, +points to a valid character, and the value returned is the number of +bytes in the character. +.It (size_t)-1 +.Fa s +points to an illegal byte sequence which does not form a valid multibyte +character in the current locale. +.Fn mbrtowc +sets +.Va errno +to EILSEQ. +The conversion state object pointed to by +.Fa mbs +is left in an undefined state and must be reinitialized before being +used again. +.Pp +Because applications using .Fn mbrtowc -returns the number of bytes in the character. +are shielded from specifics of the multibyte character encoding scheme, +it is impossible to repair byte sequences containing encoding errors. +Such byte sequences must be treated as invalid and potentially malicious input. +Applications must stop processing the byte sequence pointed to by +.Fa s +and either discard any wide characters already converted, or cope with +truncated input. .It (size_t)-2 .Fa s -points to the byte sequence which possibly contains part of a valid -multibyte character, but which is incomplete. +points to an incomplete byte sequence which contains part of a valid +multibyte character. +.Fn mbrtowc +sets +.Va errno +to EILSEQ and stores the bytes belonging to the incomplete sequence in +.Fa mbs . +The character may be completed by calling +.Fn mbrtowc +again with +.Fa s +pointing to one or more subsequent bytes of the multibyte character and +.Fa mbs +pointing to the conversion state object used during conversion of the +incomplete byte sequence. +.Pp When .Fa n is at least .Dv MB_CUR_MAX -only occurs if the array pointed to by +this situation only occurs if the byte string pointed to by .Fa s contains a redundant shift sequence. -.It (size_t)-1 -.Fa s -points to an illegal byte sequence which does not form a valid multibyte -character. -In this case, -.Fn mbrtowc -sets -.Va errno -to indicate the error. .El -.\" ---------------------------------------------------------------------- .Sh ERRORS The .Fn mbrtowc @@ -176,15 +229,13 @@ may causes an error in the following cas .Fa s points to an invalid or incomplete multibyte character. .It Bq Er EINVAL -.Fa ps +.Fa mbs points to an invalid or uninitialized mbstate_t object. .El -.\" ---------------------------------------------------------------------- .Sh SEE ALSO .Xr mbrlen 3 , .Xr mbtowc 3 , .Xr setlocale 3 -.\" ---------------------------------------------------------------------- .Sh STANDARDS The .Fn mbrtowc @@ -196,3 +247,40 @@ The restrict qualifier is added at .\" .St -isoC99 . ISO/IEC 9899:1999 .Pq Dq ISO C99 . +.Sh CAVEATS +.Fn mbrtowc +is not suitable for use by programs that care about internals of +character encoding schemes, and is cumbersome to use in programs +that need to deal with multiple character encoding schemes. +.Pp +It is possible that +.Fn mbrtowc +fails because of locale configuration errors. +An +.Dq invalid +character sequence may simply be encoded in a different encoding than that +of the current locale. +.Pp +The special cases for +.Fa s +== NULL and +.Fa mbs +== NULL do not make any sense. +Instead of passing NULL for +.Fa mbs , +.Xr mbtowc(3) +can be used. +.Pp +Earlier versions of this man page were implying that calling +.Fn mbrtowc +with a NULL +.Fa s +argument would always set +.Fa mbs +to the initial conversion state (and shift state, if applicable). +But this is true only if the previous call to +.Fn mbrtowc +using +.Fa mbs +did not return (size_t)-1 or (size_t)-2. +It is recommended to zero the mbstate_t object instead.
