YASUOKA Masahiko writes: > Hello, > > Currently pcre2 is configured with "--enable-newline-is-any". With > the option, the library treats 0x85 as a newline char. But in UTF-8, > 0x85 is used at least for some casual Kanji chars. So the pcre2 > cannot handle text which includes such the chars properly. > > Since --enable-newline-is-any conflicts with using UTF-8, I think we > should change it to --enable-newline-is-anycrlf to avoid the > conflict. > > https://github.com/PCRE2Project/pcre2/blob/pcre2-10.37/src/pcre2_internal.h#L663 > 657 /* In ASCII/Unicode, linefeed is '\n' and we equate this to NL for > 658 compatibility. NEL is the Unicode newline character; make sure it is > 659 a positive value. */ > 660 > 661 #define CHAR_LF '\n' > 662 #define CHAR_NL CHAR_LF > -> 663 #define CHAR_NEL ((unsigned char)'\x85') > 664 #define CHAR_ESC '\033' > 665 #define CHAR_DEL '\177' > 666 #define CHAR_NBSP ((unsigned char)'\xa0') > > \u8005 is "\xe0\x80\x85" in UTF-8, which includes "\x85". > https://glyphwiki.org/wiki/u8005 > > test code in php: > > <?php > $test = "\u{8005} hogehoge"; > if (preg_match("/^(.+)$/m", $test, $match)) { > print("result: " . str_ends_with($match[1], "hoge") . > " (should be 1)\n"); > } > ?> > > ok?
Good counterexample. Seems like a better default than I initially thought. ok namn@ > > > Specify --enable-newline-is-anycrlf instead of --enable-newline-is-any > which doesn't work properly with UTF-8 text. The former option treats > 0x85, which is used for some kanji in UTF-8, as a newline char.w > > Index: devel/pcre2/Makefile > =================================================================== > RCS file: /cvs/ports/devel/pcre2/Makefile,v > retrieving revision 1.16 > diff -u -p -r1.16 Makefile > --- devel/pcre2/Makefile 11 Mar 2022 18:52:29 -0000 1.16 > +++ devel/pcre2/Makefile 2 Nov 2022 14:02:31 -0000 > @@ -9,6 +9,8 @@ SHARED_LIBS += pcre2-posix > > CATEGORIES = devel > > +REVISION = 0 > + > MASTER_SITES = https://ftp.pcre.org/pub/pcre/ \ > ${MASTER_SITE_SOURCEFORGE:=pcre/} \ > http://ftp.csx.cam.ac.uk/pub/software/programming/pcre/ \ > @@ -27,7 +29,7 @@ LIB_DEPENDS = archivers/bzip2 > CONFIGURE_STYLE = gnu > CONFIGURE_ARGS = --enable-pcre2-16 \ > --enable-pcre2-32 \ > - --enable-newline-is-any \ > + --enable-newline-is-anycrlf \ > --enable-pcre2grep-libz \ > --enable-pcre2grep-libbz2 \ > --enable-pcre2test-libreadline
