Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-13 Thread Nick Coghlan
On 11 Dec. 2017 6:50 am, "INADA Naoki" wrote: Except one typo I commented on Github, I accept PEP 540. Well done, Victor and Nick for PEP 540 and 538. Python 3.7 will be most UTF-8 friendly Python 3 than ever. And thank you for all of your work on reviewing them! The appropriate trade-offs bet

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-11 Thread Guido van Rossum
Congrats Victor! Thanks mr. Inada for reviewing this PEP (and 538). Thanks everyone else who participated in the lively discussion! On Sun, Dec 10, 2017 at 4:00 PM, INADA Naoki wrote: > > > > Could you explain why not? utf-8 seems like the common thread for using > > surrogateescape so I'm not s

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-10 Thread INADA Naoki
> > Could you explain why not? utf-8 seems like the common thread for using > surrogateescape so I'm not sure what would make en_US.UTF-8 different than > C.UTF-8. > Because there are many lang_COUNTRY.UTF-8 locales: ja_JP.UTF-8, zh_TW.UTF-8, fr_FR.UTF-8, etc... If only en_US.UTF-8 should use sur

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-10 Thread Toshio Kuratomi
On Dec 9, 2017 8:53 PM, "INADA Naoki" wrote: > Earlier versions of PEP 538 thus included "en_US.UTF-8" on the > candidate target locale list, but that turned out to cause assorted > problems due to the "C -> en_US" part of the coercion. Hm, but PEP 538 says: > this PEP instead proposes to exten

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-10 Thread Victor Stinner
2017-12-10 18:46 GMT+01:00 INADA Naoki : > Except one typo I commented on Github, Fixed: https://github.com/python/peps/commit/08224bf6bdf16b539fb6f8136061877e5924476d > I accept PEP 540. Wow, thank you :-) Again, thank you for your very useful feedback which helped to make the PEP 540 much bet

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-10 Thread INADA Naoki
Except one typo I commented on Github, I accept PEP 540. Well done, Victor and Nick for PEP 540 and 538. Python 3.7 will be most UTF-8 friendly Python 3 than ever. INADA Naoki On Mon, Dec 11, 2017 at 2:21 AM, Victor Stinner wrote: > Ok, I fixed the effects of the locale coercion (PEP 538). D

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-10 Thread Victor Stinner
Ok, I fixed the effects of the locale coercion (PEP 538). Does it now look good to you, Naoki? https://www.python.org/dev/peps/pep-0540/#relationship-with-the-locale-coercion-pep-538 The commit: https://github.com/python/peps/commit/71cda51fbb622ece63f7a9d3c8fa6cd33ce06b58 diff --git a/pep-0540

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-10 Thread Victor Stinner
Hi, Le 10 déc. 2017 05:48, "INADA Naoki" a écrit : Now I'm OK to accept the PEP, except one nitpick. I got a private email about the same issue. I don't think that it's nitpicking since many people were confused about the relationship between the PEP 538 and PEP 540. So it seems like I was con

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-09 Thread INADA Naoki
> Earlier versions of PEP 538 thus included "en_US.UTF-8" on the > candidate target locale list, but that turned out to cause assorted > problems due to the "C -> en_US" part of the coercion. Hm, but PEP 538 says: > this PEP instead proposes to extend the "surrogateescape" default for stdin > an

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-09 Thread INADA Naoki
Now I'm OK to accept the PEP, except one nitpick. > > Locale coercion only impacts non-Python code like C libraries, whereas > the Python UTF-8 Mode only impacts Python code: the two PEPs are > complementary. > This sentence seems bit misleading. If UTF-8 mode is disabled explicitly, locale coerc

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Nick Coghlan
On 9 December 2017 at 01:22, Victor Stinner wrote: > I updated my PEP: in the 4th version, locale.getpreferredencoding() > now returns 'UTF-8' in the UTF-8 Mode. +1, that's a good change, since it brings the "locale coercion failed" case even closer to the "locale coercion succeeded" behaviour.

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Victor Stinner
2017-12-08 17:29 GMT+01:00 Ethan Furman : > For those of us trying to follow along, is this change to open() one that > Inada-san was worried about? Has something else changed? I agree that my PEP is evolving quickly, that's why I added a "Version History" at the end: https://www.python.org/dev/p

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Ethan Furman
There were some concerns about open() earlier: On Wed, 6 Dec 2017 at 06:10 INADA Naoki wrote: > I think PEP 538 and PEP 540 should behave almost identical except > changing locale or not. So I need very strong reason if PEP 540 > changes default error handler of open(). Brett replied: > I don't

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Victor Stinner
2017-12-08 16:22 GMT+01:00 Victor Stinner : > I updated my PEP: in the 4th version, locale.getpreferredencoding() > now returns 'UTF-8' in the UTF-8 Mode. Sorry, I forgot to mention that I already updated the implementation to the latest version of the PEP: https://github.com/python/cpython/pull/8

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Victor Stinner
I updated my PEP: in the 4th version, locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8 Mode. https://www.python.org/dev/peps/pep-0540/ I also clarified the direct effects of the UTF-8 Mode, but also listed the most user visible changes as "Side effects". """ Effects of the UTF-8 Mo

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Victor Stinner
2017-12-08 15:01 GMT+01:00 INADA Naoki : >> In short, locale coercion and UTF-8 mode will be both enabled by the >> POSIX locale. > > Hm, it is bit surprising because I thought UTF-8 mode is fallback > of locale coercion when coercion is failed or disabled. I rewrote the "differences between the P

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread INADA Naoki
On Fri, Dec 8, 2017 at 7:22 PM, Victor Stinner wrote: >> >> Both of PEP 538 (locale coercion) and PEP 540 (UTF-8 mode) shares >> same logic to detect POSIX locale. >> >> When POSIX locale is detected, locale coercion is tried first. And if >> locale coercion >> succeeds, UTF-8 mode is not used be

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Victor Stinner
2017-12-08 6:11 GMT+01:00 INADA Naoki : > Or should we change loale.getpreferredencoding() to return UTF-8 > instead of ASCII always, regardless of PEP 538 and 540? On the POSIX locale, if the locale coercion works (PEP 538), locale.getpreferredencoding() returns UTF-8. We are good. The question

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 Thread Victor Stinner
Hi, Oh, locale.getpreferredencoding(), that's a good question :-) 2017-12-08 6:02 GMT+01:00 INADA Naoki : > But I want to clarify more about difference/relationship between PEP > 538 and 540. > > If I understand correctly: > > Both of PEP 538 (locale coercion) and PEP 540 (UTF-8 mode) shares > sa

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-07 Thread INADA Naoki
> Or locale.getpreferredencoding() returns UTF-8 in UTF-8 mode too? Or should we change loale.getpreferredencoding() to return UTF-8 instead of ASCII always, regardless of PEP 538 and 540? INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-07 Thread INADA Naoki
Looks nice. But I want to clarify more about difference/relationship between PEP 538 and 540. If I understand correctly: Both of PEP 538 (locale coercion) and PEP 540 (UTF-8 mode) shares same logic to detect POSIX locale. When POSIX locale is detected, locale coercion is tried first. And if loc

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-07 Thread Chris Barker - NOAA Federal
I made the following two changes to the PEP 540: * open() error handler remains "strict" * remove the "Strict UTF8 mode" which doesn't make much sense anymore +1 — ignore my previous note. -CHB I wrote the Strict UTF-8 mode when open() used surrogateescape error handler in the UTF-8 mode. I d

[Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-07 Thread Victor Stinner
Hi, I made the following two changes to the PEP 540: * open() error handler remains "strict" * remove the "Strict UTF8 mode" which doesn't make much sense anymore I wrote the Strict UTF-8 mode when open() used surrogateescape error handler in the UTF-8 mode. I don't think that a Strict UTF-8 mod