Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

On 11 Dec. 2017 6:50 am, "INADA Naoki" wrote: Except one typo I commented on Github, I accept PEP 540. Well done, Victor and Nick for PEP 540 and 538. Python 3.7 will be most UTF-8 friendly Python 3 than ever. And thank you for all of your work on reviewing them! The appropriate trade-offs bet

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

Congrats Victor! Thanks mr. Inada for reviewing this PEP (and 538). Thanks everyone else who participated in the lively discussion! On Sun, Dec 10, 2017 at 4:00 PM, INADA Naoki wrote: > > > > Could you explain why not? utf-8 seems like the common thread for using > > surrogateescape so I'm not s

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

> > Could you explain why not? utf-8 seems like the common thread for using > surrogateescape so I'm not sure what would make en_US.UTF-8 different than > C.UTF-8. > Because there are many lang_COUNTRY.UTF-8 locales: ja_JP.UTF-8, zh_TW.UTF-8, fr_FR.UTF-8, etc... If only en_US.UTF-8 should use sur

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

On Dec 9, 2017 8:53 PM, "INADA Naoki" wrote: > Earlier versions of PEP 538 thus included "en_US.UTF-8" on the > candidate target locale list, but that turned out to cause assorted > problems due to the "C -> en_US" part of the coercion. Hm, but PEP 538 says: > this PEP instead proposes to exten

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-10 18:46 GMT+01:00 INADA Naoki : > Except one typo I commented on Github, Fixed: https://github.com/python/peps/commit/08224bf6bdf16b539fb6f8136061877e5924476d > I accept PEP 540. Wow, thank you :-) Again, thank you for your very useful feedback which helped to make the PEP 540 much bet

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

Except one typo I commented on Github, I accept PEP 540. Well done, Victor and Nick for PEP 540 and 538. Python 3.7 will be most UTF-8 friendly Python 3 than ever. INADA Naoki On Mon, Dec 11, 2017 at 2:21 AM, Victor Stinner wrote: > Ok, I fixed the effects of the locale coercion (PEP 538). D

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

Ok, I fixed the effects of the locale coercion (PEP 538). Does it now look good to you, Naoki? https://www.python.org/dev/peps/pep-0540/#relationship-with-the-locale-coercion-pep-538 The commit: https://github.com/python/peps/commit/71cda51fbb622ece63f7a9d3c8fa6cd33ce06b58 diff --git a/pep-0540

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

Hi, Le 10 déc. 2017 05:48, "INADA Naoki" a écrit : Now I'm OK to accept the PEP, except one nitpick. I got a private email about the same issue. I don't think that it's nitpicking since many people were confused about the relationship between the PEP 538 and PEP 540. So it seems like I was con

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

> Earlier versions of PEP 538 thus included "en_US.UTF-8" on the > candidate target locale list, but that turned out to cause assorted > problems due to the "C -> en_US" part of the coercion. Hm, but PEP 538 says: > this PEP instead proposes to extend the "surrogateescape" default for stdin > an

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

Now I'm OK to accept the PEP, except one nitpick. > > Locale coercion only impacts non-Python code like C libraries, whereas > the Python UTF-8 Mode only impacts Python code: the two PEPs are > complementary. > This sentence seems bit misleading. If UTF-8 mode is disabled explicitly, locale coerc

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

On 9 December 2017 at 01:22, Victor Stinner wrote: > I updated my PEP: in the 4th version, locale.getpreferredencoding() > now returns 'UTF-8' in the UTF-8 Mode. +1, that's a good change, since it brings the "locale coercion failed" case even closer to the "locale coercion succeeded" behaviour.

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 17:29 GMT+01:00 Ethan Furman : > For those of us trying to follow along, is this change to open() one that > Inada-san was worried about? Has something else changed? I agree that my PEP is evolving quickly, that's why I added a "Version History" at the end: https://www.python.org/dev/p

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

There were some concerns about open() earlier: On Wed, 6 Dec 2017 at 06:10 INADA Naoki wrote: > I think PEP 538 and PEP 540 should behave almost identical except > changing locale or not. So I need very strong reason if PEP 540 > changes default error handler of open(). Brett replied: > I don't

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 16:22 GMT+01:00 Victor Stinner : > I updated my PEP: in the 4th version, locale.getpreferredencoding() > now returns 'UTF-8' in the UTF-8 Mode. Sorry, I forgot to mention that I already updated the implementation to the latest version of the PEP: https://github.com/python/cpython/pull/8

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

I updated my PEP: in the 4th version, locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8 Mode. https://www.python.org/dev/peps/pep-0540/ I also clarified the direct effects of the UTF-8 Mode, but also listed the most user visible changes as "Side effects". """ Effects of the UTF-8 Mo

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 15:01 GMT+01:00 INADA Naoki : >> In short, locale coercion and UTF-8 mode will be both enabled by the >> POSIX locale. > > Hm, it is bit surprising because I thought UTF-8 mode is fallback > of locale coercion when coercion is failed or disabled. I rewrote the "differences between the P

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

On Fri, Dec 8, 2017 at 7:22 PM, Victor Stinner wrote: >> >> Both of PEP 538 (locale coercion) and PEP 540 (UTF-8 mode) shares >> same logic to detect POSIX locale. >> >> When POSIX locale is detected, locale coercion is tried first. And if >> locale coercion >> succeeds, UTF-8 mode is not used be

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

2017-12-08 6:11 GMT+01:00 INADA Naoki : > Or should we change loale.getpreferredencoding() to return UTF-8 > instead of ASCII always, regardless of PEP 538 and 540? On the POSIX locale, if the locale coercion works (PEP 538), locale.getpreferredencoding() returns UTF-8. We are good. The question

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

Hi, Oh, locale.getpreferredencoding(), that's a good question :-) 2017-12-08 6:02 GMT+01:00 INADA Naoki : > But I want to clarify more about difference/relationship between PEP > 538 and 540. > > If I understand correctly: > > Both of PEP 538 (locale coercion) and PEP 540 (UTF-8 mode) shares > sa

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

Victor Stinner wrote: Users don't use stdin and stdout as regular files, they are more used as pipes to pass data between programs with the Unix pipe in a shell like "producer | consumer". Sometimes stdout is redirected to a file, but I consider that it is expected to behave as a pipe and the reg

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

> Or locale.getpreferredencoding() returns UTF-8 in UTF-8 mode too? Or should we change loale.getpreferredencoding() to return UTF-8 instead of ASCII always, regardless of PEP 538 and 540? INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

ed in Python internals and ignores the > locale: non-Python running in the same process is not aware of the > "Python UTF-8 mode". The benefit of the PEP 538 approach is that it helps > ensure that encoding handling in binary extension modules and subprocesses > is consistent wit

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

Windows filesystem encoding to UTF-8" * `PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_: "Change Windows console encoding to UTF-8" * `PEP 383 <https://www.python.org/dev/peps/pep-0383/>`_: "Non-decodable Bytes in System Character Interfaces" Post His

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

I’m a bit confused: File names and the like are one thing, and the CONTENTS of files is quite another. I get that there is theoretically a “default” encoding for the contents of text files, but that is SO likely to be wrong as to be ignorable. open() already defaults to utf-8. Which is a fine de

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 12/7/2017 5:45 PM, Jonathan Goble wrote: On Thu, Dec 7, 2017 at 8:38 PM Glenn Linderman > wrote: If it were to be changed, one could add a text-mode option in 3.7, say "t" in the mode string, and a PendingDeprecationWarning for open calls without th

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On Thu, Dec 7, 2017 at 8:38 PM Glenn Linderman wrote: > If it were to be changed, one could add a text-mode option in 3.7, say "t" > in the mode string, and a PendingDeprecationWarning for open calls without > the specification of either t or b in the mode string. > "t" is already supported in o

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 12/7/2017 4:48 PM, Victor Stinner wrote: Ok, now comes the real question, open(). For open(), I used the example of a code snippet *writing* the content of a directory (os.listdir) into a text file. Another example is to read filenames from a text files but pass-through undecodable bytes tha

[Python-Dev] PEP 540: Add a new UTF-8 mode (v3)

to UTF-8" * `PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_: "Change Windows console encoding to UTF-8" * `PEP 383 <https://www.python.org/dev/peps/pep-0383/>`_: "Non-decodable Bytes in System Character Interfaces" Post History

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

2017-12-08 0:26 GMT+01:00 Guido van Rossum : > You will quickly get decoding errors, and that is INADA's point. (Unless you > use encoding='Latin-1'.) His worry is that the surrogateescape error handler > makes it so that you won't get decoding errors, and then the failure mode is > much harder to

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On Thu, Dec 7, 2017 at 3:02 PM, Victor Stinner wrote: > 2017-12-06 5:07 GMT+01:00 INADA Naoki : > > And opening binary file without "b" option is very common mistake of new > > developers. If default error handler is surrogateescape, they lose a > chance > > to notice their bug. > > To come back

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

2017-12-06 5:07 GMT+01:00 INADA Naoki : > And opening binary file without "b" option is very common mistake of new > developers. If default error handler is surrogateescape, they lose a chance > to notice their bug. To come back to your original point, I didn't know that it was a common mistake t

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

While I'm not strongly convinced that open() error handler must be changed for surrogateescape, first I would like to make sure that it's really a very bad idea because changing it :-) 2017-12-07 7:49 GMT+01:00 INADA Naoki : > I just came up with crazy idea; changing default error handler of open

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

> I care only about builtin open()'s behavior. > PEP 538 doesn't change default error handler of open(). > > I think PEP 538 and PEP 540 should behave almost identical except > changing locale > or not. So I need very strong reason if PEP 540 changes default error > handler of open(). > I just ca

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 7 December 2017 at 08:20, Victor Stinner wrote: > 2017-12-06 23:07 GMT+01:00 Antoine Pitrou : >> One question: how do you plan to test for the POSIX locale? > > I'm not sure. I will probably rely on Nick for that ;-) Nick already > implemented this exact check for his PEP 538 which is already >

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 7 December 2017 at 01:59, Jakub Wilk wrote: > * Nick Coghlan , 2017-12-06, 16:15: >> The one that's relevant to default locale detection is just the string >> that "setlocale(LC_CTYPE, NULL)" returns. > > POSIX doesn't require any particular return value for setlocale() calls. > It's only guara

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On Thu, 7 Dec 2017 00:22:52 +0100 Victor Stinner wrote: > 2017-12-06 23:36 GMT+01:00 Antoine Pitrou : > > Other than that, +1 on the PEP. > > Naoki doesn't seem to be confortable with the usage of the > surrogateescape error handler by default for open(). Are you ok with > that? If yes, would y

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

2017-12-06 23:36 GMT+01:00 Antoine Pitrou : > Other than that, +1 on the PEP. Naoki doesn't seem to be confortable with the usage of the surrogateescape error handler by default for open(). Are you ok with that? If yes, would you mind to explain why? :-) Victor ___

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On Wed, 6 Dec 2017 23:20:41 +0100 Victor Stinner wrote: > 2017-12-06 23:07 GMT+01:00 Antoine Pitrou : > > One question: how do you plan to test for the POSIX locale? > > I'm not sure. I will probably rely on Nick for that ;-) Nick already > implemented this exact check for his PEP 538 which is

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

2017-12-06 23:07 GMT+01:00 Antoine Pitrou : > One question: how do you plan to test for the POSIX locale? I'm not sure. I will probably rely on Nick for that ;-) Nick already implemented this exact check for his PEP 538 which is already implemented in Python 3.7. I already implemented the PEP 540

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On Wed, 6 Dec 2017 01:49:41 +0100 Victor Stinner wrote: > Hi, > > I knew that I had to rewrite my PEP 540, but I was too lazy. Since > Guido explicitly requested a shorter PEP, here you have! > > https://www.python.org/dev/peps/pep-0540/ > > Trust me, it's the same PEP, but focused on the most

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

Victor Stinner wrote: Maybe the "UTF-8 Mode" should be renamed to "UTF-8 with surrogateescape, or backslashreplace for stderr, or surrogatepass for fsencode/fsencode on Windows, or strict for Strict UTF-8 Mode"... But the PEP title would be too long, no? :-) Relaxed UTF-8 Mode? UTF8-Yeah-I'm-F

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On Wed, 6 Dec 2017 at 06:10 INADA Naoki wrote: > >> And I have one worrying point. > >> With UTF-8 mode, open()'s default encoding/error handler is > >> UTF-8/surrogateescape. > > > > The Strict UTF-8 Mode is for you if you prioritize correctness over > usability. > > Yes, but as I said, I cares

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

* Nick Coghlan , 2017-12-06, 16:15: Something I've just noticed that needs to be clarified: on Linux, "C" locale and "POSIX" locale are aliases, but this isn't true in general (e.g. it's not the case on *BSD systems, including Mac OS X). For those of us with little to no BSD/MacOS experience, ca

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

>> And I have one worrying point. >> With UTF-8 mode, open()'s default encoding/error handler is >> UTF-8/surrogateescape. > > The Strict UTF-8 Mode is for you if you prioritize correctness over usability. Yes, but as I said, I cares about not experienced developer who doesn't know what UTF-8 mode

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 6 December 2017 at 20:38, Victor Stinner wrote: > Nick: >> So if PEP 540 is going to implicitly trigger switching encodings, it >> needs to specify whether it's going to look for the C locale or the >> POSIX locale (I'd suggest C locale, since that's the actual default >> that causes problems).

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

Nick: > So if PEP 540 is going to implicitly trigger switching encodings, it > needs to specify whether it's going to look for the C locale or the > POSIX locale (I'd suggest C locale, since that's the actual default > that causes problems). I'm thinking at the test already used by check_force_asc

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

Hi Naoki, 2017-12-06 5:07 GMT+01:00 INADA Naoki : > Oh, revised version is really short! > > And I have one worrying point. > With UTF-8 mode, open()'s default encoding/error handler is > UTF-8/surrogateescape. The Strict UTF-8 Mode is for you if you prioritize correctness over usability. In the

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 6 December 2017 at 16:18, Glenn Linderman wrote: > "b" mostly matters on Windows, correct? And Windows doesn't use C or POSIX > locale, correct? And if these are correct, then is this an issue? And if so, > why? In Python 3, "b" matters everywhere, since it controls whether the stream gets wra

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 12/5/2017 8:07 PM, INADA Naoki wrote: Oh, revised version is really short! And I have one worrying point. With UTF-8 mode, open()'s default encoding/error handler is UTF-8/surrogateescape. Containers are really growing. PyCharm supports Docker and many new Python developers use Docker inste

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 6 December 2017 at 15:59, Chris Angelico wrote: > On Wed, Dec 6, 2017 at 4:46 PM, Nick Coghlan wrote: >> Something I've just noticed that needs to be clarified: on Linux, "C" >> locale and "POSIX" locale are aliases, but this isn't true in general >> (e.g. it's not the case on *BSD systems, in

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On Wed, Dec 6, 2017 at 4:46 PM, Nick Coghlan wrote: > Something I've just noticed that needs to be clarified: on Linux, "C" > locale and "POSIX" locale are aliases, but this isn't true in general > (e.g. it's not the case on *BSD systems, including Mac OS X). For those of us with little to no BSD

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

Something I've just noticed that needs to be clarified: on Linux, "C" locale and "POSIX" locale are aliases, but this isn't true in general (e.g. it's not the case on *BSD systems, including Mac OS X). To handle that in PEP 538, I made it clear that everything is keyed specifically off the "C" loc

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

Oh, revised version is really short! And I have one worrying point. With UTF-8 mode, open()'s default encoding/error handler is UTF-8/surrogateescape. Containers are really growing. PyCharm supports Docker and many new Python developers use Docker instead of installing Python directly on their s

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

I'm sorry about my laziness. I've very busy these months, but I'm back to OSS world from today. While I should review carefully again, I think I'm close to accept PEP 540. * PEP 540 really helps containers and old Linux machines PEP 538 doesn't work. And containers is really important for these

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

On 6 December 2017 at 11:01, Victor Stinner wrote: >> Annex: Differences between the PEP 538 and the PEP 540 >> == >> >> The PEP 538 uses the "C.UTF-8" locale which is quite new and only >> supported by a few Linux distributions; this locale is n

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

> Annex: Differences between the PEP 538 and the PEP 540 > == > > The PEP 538 uses the "C.UTF-8" locale which is quite new and only > supported by a few Linux distributions; this locale is not currently > supported by FreeBSD or macOS for example.

[Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

`PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_: "Change Windows console encoding to UTF-8" * `PEP 383 <https://www.python.org/dev/peps/pep-0383/>`_: "Non-decodable Bytes in System Character Interfaces" Post History * 2017-12: `[Python-Dev] PEP 540

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

Chris: > I just took another look at 538 -- and yes, the relationship between the two > is really unclear. In particular, with 538, why do we need 540? I honestly > don't know. The PEP 538 only impacts platforms which provide the C.UTF-8 locale or a variant: only a few recent Linux distribution. I

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

On Tue, Dec 5, 2017 at 1:18 PM, Guido van Rossum wrote: > > > I am very worried about this long and rambling PEP, > FWIW, I read the PEP on the bus this morning on a phone, and while lng, I didn't find it too rambling. And this topic has been very often discussed in very long and rambling mailing

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

2017-12-05 22:18 GMT+01:00 Guido van Rossum : > So I guess PEP acceptance week is over. :-( My bad, Barry wrote "PEP Acceptance Day", not week, on twitter ;-) https://twitter.com/pumpichank/status/937770805076905990 Victor ___ Python-Dev mailing list Py

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

I've been discussing this PEP offline with Victor, but he suggested we should discuss it in public instead. I am very worried about this long and rambling PEP, and I propose that it not be accepted without a major rewrite to focus on clarity of the specification. The "Unicode just works" summary i

[Python-Dev] PEP 540: Add a new UTF-8 mode

Hi, Since it's the PEP Acceptance Week, I try my luck! Here is my very long PEP to propose a tiny change. The PEP is very long to explain the rationale and limitations. Inaccurate tl; dr with the UTF-8 mode, Unicode "just works" as expected. Reminder: INADA Naoki was nominated as the BDFL-Delega