Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale

[INADA Naoki]

> It seems to based on an assumption that the C locale is always some kind of
> pathology. Admittedly, it sometimes is a result of misconfiguration or a
> mistake. (But I don't see why it's the interpreter's job to correct such
> mistakes.) However, in some cases the C locale is a normal environment for
> system services, cron scripts, distro package builds and whatnot.

I think "C locale + use UTF-8 for stdio + fs" is common setup,
especially for servers.
It's not mistake or misconfiguration.  Perl, Ruby, Rust, Node.JS and
Go can use UTF-8
without any pain on C locale.  And current Python is painful for such cases.
So I strongly +1 for PEP 540 (UTF-8 mode).

On the other hand, PEP 538 is for for locale-dependent libraries (like
curses) and
subprocesses.
I agree C locale is misconfiguration if user want to use UTF-8 in
locale-dependent
libraries.  And I agree current PEP 538 seems carrying it a bit too far.

But locale coercing works nice on platforms like android.
So how about simplified version of PEP 538?  Just adding configure
option for locale coercing
which is disabled by default.  No envvar options and no warnings.

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale

[Nick Coghlan]

On 13 March 2017 at 18:37, INADA Naoki  wrote:

> But locale coercing works nice on platforms like android.
> So how about simplified version of PEP 538?  Just adding configure
> option for locale coercing
> which is disabled by default.  No envvar options and no warnings.
>

That doesn't solve my original Linux distro problem, where locale
misconfiguration problems show up as "Python 2 works, Python 3 doesn't
work" behaviour and bug reports.

The problem is that where Python 2 was largely locale-independent by
default (just passing raw bytes through) such that you'd only get immediate
encoding or decoding errors if you had a Unicode literal or a decode() call
somewhere in your code and would otherwise pass data corruption problems
further down the chain, Python 3 is locale-*aware* by default, and eagerly
decodes:

- command line parameters
- environment variables
- responses from operating system API calls
- standard stream input
- file contents

You *can* still write locale-independent Python 3 applications, but they
involve sprinkling liberal doses of "b" prefixes and suffixes and mode
settings and "surrogateescape" error handler declarations in various places
- you can't just run python-modernize over a pre-existing Python 2
application and expect it to behave the same way in the C locale as it did
before.

Once implemented, PEP 540 will partially solve the problem by introducing a
locale independent UTF-8 mode, but that still leaves the inconsistency with
other locale-aware components that are needing to deal with Python 3 API
calls that accept or return Unicode objects where Python 2 allowed the use
of 8-bit strings.

Folks that really want the old behaviour back will be able to set
PYTHONCOERCECLOCALE=0 (as that no longer emits any warnings), or else build
their own CPython from source using `--without-c-locale-coercion` and
``--without-c-locale-warning`. However, they'll also get the explicit
support notification from PEP 11 that any Unicode handling bugs they run
into in those configurations are entirely their own problem - we won't fix
them, because we consider those configurations unsupportable in the general
case.

That puts the additional self-support burden on folks doing something
unusual (i.e. insisting on running an ASCII-only environment in 2017),
rather than on those with a more conventional use case (i.e. running an up
to date \*nix OS using UTF-8 or another universal encoding for both local
and remote interfaces).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale

[INADA Naoki]

On Mon, Mar 13, 2017 at 8:01 PM, Nick Coghlan  wrote:
> On 13 March 2017 at 18:37, INADA Naoki  wrote:
>>
>> But locale coercing works nice on platforms like android.
>> So how about simplified version of PEP 538?  Just adding configure
>> option for locale coercing
>> which is disabled by default.  No envvar options and no warnings.
>
>
> That doesn't solve my original Linux distro problem, where locale
> misconfiguration problems show up as "Python 2 works, Python 3 doesn't work"
> behaviour and bug reports.

Sorry, I meant "PEP 540 + Simplified PEP 538 (coercing by configure option)".
distros can enable the configure option, off course.


>
> The problem is that where Python 2 was largely locale-independent by default
> (just passing raw bytes through) such that you'd only get immediate encoding
> or decoding errors if you had a Unicode literal or a decode() call somewhere
> in your code and would otherwise pass data corruption problems further down
> the chain, Python 3 is locale-*aware* by default, and eagerly decodes:
>
> - command line parameters
> - environment variables
> - responses from operating system API calls
> - standard stream input
> - file contents
>
> You *can* still write locale-independent Python 3 applications, but they
> involve sprinkling liberal doses of "b" prefixes and suffixes and mode
> settings and "surrogateescape" error handler declarations in various places
> - you can't just run python-modernize over a pre-existing Python 2
> application and expect it to behave the same way in the C locale as it did
> before.
>
> Once implemented, PEP 540 will partially solve the problem by introducing a
> locale independent UTF-8 mode, but that still leaves the inconsistency with
> other locale-aware components that are needing to deal with Python 3 API
> calls that accept or return Unicode objects where Python 2 allowed the use
> of 8-bit strings.

I feel problems PEP 538 solves, but PEP 540 doesn't solve are relatively small
compared with complexity introduced PEP 538.  As my understanding, PEP 538
solves problems only when:

* python executable is used.  (GUI applications linking Python for
plugin is not affected)
* One of C.UTF-8, C.utf8 or UTF8 is accepted for LC_CTYPE.
* The "locale aware components" uses something other than ASCII or
UTF-8 on C locale,
   but uses UTF-8 on UTF-8 locale.

Can't we reduce options from 3 (2 configure, 1 envvar) when PEP 540 is
accepted too?


>
> Folks that really want the old behaviour back will be able to set
> PYTHONCOERCECLOCALE=0 (as that no longer emits any warnings), or else build
> their own CPython from source using `--without-c-locale-coercion` and
> ``--without-c-locale-warning`. However, they'll also get the explicit
> support notification from PEP 11 that any Unicode handling bugs they run
> into in those configurations are entirely their own problem - we won't fix
> them, because we consider those configurations unsupportable in the general
> case.
>
> That puts the additional self-support burden on folks doing something
> unusual (i.e. insisting on running an ASCII-only environment in 2017),
> rather than on those with a more conventional use case (i.e. running an up
> to date \*nix OS using UTF-8 or another universal encoding for both local
> and remote interfaces).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale

[Random832]

On Mon, Mar 13, 2017, at 04:37, INADA Naoki wrote:
> But locale coercing works nice on platforms like android.
> So how about simplified version of PEP 538?  Just adding configure
> option for locale coercing
> which is disabled by default.  No envvar options and no warnings.

A configure option just kicks the decision to packagers - either no-one
uses it (and thus it solves nothing) or people do use it (and any
problems it causes won't be mitigated at all)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale

[INADA Naoki]

On Mon, Mar 13, 2017 at 10:31 PM, Random832  wrote:
> On Mon, Mar 13, 2017, at 04:37, INADA Naoki wrote:
>> But locale coercing works nice on platforms like android.
>> So how about simplified version of PEP 538?  Just adding configure
>> option for locale coercing
>> which is disabled by default.  No envvar options and no warnings.
>
> A configure option just kicks the decision to packagers - either no-one
> uses it (and thus it solves nothing) or people do use it (and any
> problems it causes won't be mitigated at all)

Yes.  people who building Python understand about the platform than
users in most cases.

For android build, they know coercing is works well on android.

For Linux distros, they know the system supports locales like C.UTF-8
or not, and there are
any python- packages which may cause the problem and coercing solve it.

For people who building Python themselves (in docker, pyenv, etc...)
They knows how they
use the Python.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Regarding writing tests for module tabnanny

[Jaysinh Shukla]

Respected Members,

I identified the standard module 'tabnanny' is having 16.66% of 
code coverage (Source: 
https://codecov.io/gh/python/cpython/src/master/Lib/tabnanny.py). I am 
interested to write tests for this module. Before starting, I would like 
to get help from any core developer on this.



1. Is community expecting to have tests for this module?
2. Any module specific guidelines?

I am waiting for green signal from any core developer. Thanks!

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Regarding writing tests for module tabnanny

[Brett Cannon]

These questions are best asked on the core-mentorship mailing list,
Jaysinh, but to quickly answer your question:

1. Yes, tests would be appreciated.
2. Nothing from me

On Mon, 13 Mar 2017 at 08:20 Jaysinh Shukla  wrote:

> Respected Members,
>
>  I identified the standard module 'tabnanny' is having 16.66% of
> code coverage (Source:
> https://codecov.io/gh/python/cpython/src/master/Lib/tabnanny.py). I am
> interested to write tests for this module. Before starting, I would like
> to get help from any core developer on this.
>
>
>  1. Is community expecting to have tests for this module?
>  2. Any module specific guidelines?
>
> I am waiting for green signal from any core developer. Thanks!
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Regarding writing tests for module tabnanny

[Martin Panter]

On 13 March 2017 at 11:56, Jaysinh Shukla  wrote:
> Respected Members,
>
> I identified the standard module 'tabnanny' is having 16.66% of code
> coverage (Source:
> https://codecov.io/gh/python/cpython/src/master/Lib/tabnanny.py). I am
> interested to write tests for this module. Before starting, I would like to
> get help from any core developer on this.
>
>
> 1. Is community expecting to have tests for this module?
> 2. Any module specific guidelines?
>
> I am waiting for green signal from any core developer. Thanks!

Try the people involved in the existing patches at
 and
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com