[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Inada Naoki]

On Thu, Feb 11, 2021 at 4:44 PM Jim J. Jewett  wrote:
>
> I just reread PEP 597, then re-reread the Rationale.
>

Do you read current PEP 597, or old PEP 597 in discuss.python.org?


-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UKGDVMHUNNNRA4D4UCG4RLPZDIVKNNEY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

[Guido van Rossum]

On Thu, Feb 11, 2021 at 12:00 AM Jim J. Jewett  wrote:

> (1)  Is it really a TypeGuard, or more of a TypeAssertion?
>

It's a query, not an assertion. The same concept is called type guard in
TypeScript.


> (2)  Does this push too hard on "annotations only have one meaning"?  If
> it has to imported from typing, then probably not, but I think that is
> worth adding to the PEP.
>

No. *This* particular annotation has a meaning for static type checkers. If
you use annotations for something else, don't use it.


> (3)  Why can't the falsey case of an Optional String narrow to a set of
> literals {"", None}  Are you worried that a subclass of str might have its
> own empty string, or just that you don't want to promise this?  As written,
> it sounds like such a narrowing is forbidden.
>

 A type checker could do this, but I don't see a reason why we should
prescribe that type checkers *should* do that. Some type checkers (in
particular, mypy) are reluctant to infer union types, and there are good
reasons for this (but this margin isn't wide enough to explain it).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TZ6UZAPFX7OG64PFMQUZKDPBVYD6A5WK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

[Jim J. Jewett]

If a TypeGuard returns false, does that mean "it doesn't match", or just "I 
can't prove it matches, but it still might"?  That seems relevant to the else 
clause ... and seems to have changed since the last version I looked at.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KCJ2IIF4BUXGQEBLBPFNKPXZCGUHVIPD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Jim J. Jewett]

Inada Naoki wrote:

> Default encoding is used for:

> a. Really need to use locale specific encoding
> b. UTF-8 (bug. not work on Windows)
> c. ASCII (not a bug, but slow on Windows)

> I assume most usages are (b) and (c). This PEP can reduce them soon.

Is this just an assumption, based on those times being visible to someone who 
installs a lot of packages, or has the use of any locale other than UTF-8 and 
ASCII really gone down a lot?  Have browsers stopped using charset sniffing?

> Additionally, encoding="locale" will be backward/forward compatible

What would be the problem with changing the default from None to locale?  (I 
think you mentioned that they are the same 99% of the time; is that other 1% 
likely to be cases where locale is wrong but None is right?  Would there be a 
better way to represent that 1%?)

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NGAF753ALAPMUKNJWFBYLDOTYTUJH6ZG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Security releases of CPython

[Michał Górny]

Hello,

I'm the primary maintainer of CPython packages in Gentoo. I would like
to discuss possible improvement to the release process in order to
accelerate releasing security fixes to users.


I feel that vulnerability fixes do not make it to end users fast enough.
For example, according to the current release schedules for 3.9 and 3.8,
the bugfix releases are planned two months apart. While the release is
expected to happen in the next few days, both versions are known to be
vulnerable for almost a month!

Ironically, things look even worse for security-supported versions.
Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
schedule (planned for Jan 15th), and they are known to be vulnerable
since mid-October.

In my opinion, this causes three issues:

1. Users using official releases are exposed to security vulnerabilities
for prolonged periods of time.

2. When releases happen, security fixes are often combined with many
other changes. This causes problems for distribution maintainers who, on
one hand, would like to deploy the security fixes to production versions
ASAP, and on the other, would prefer that the new version remained in
testing for some time due to the other changes.

3. Effectively, it means that distribution developers need to track
and backport security fixes themselves. In the end, this means a lot of
duplicate work.


I think that security fixes are important enough to justify not sticking
to a strict release schedule. Therefore, I would like to propose that if
vulnerability fixes are committed, new releases are made
as frequently as necessary and as soon as possible (i.e. possibly giving
some time for testing) rather than according to a strict schedule.

Furthermore, I think that at least for branches that are in higher level
of maintenance than security, it could make sense to actually make
security releases (e.g. 3.9.1.x) that would include only security fixes
without other changes.


What do you think?

-- 
Best regards,
Michał Górny
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VKIRQTSQCO3MTXS57MDGJE6FH3PJCEBI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Jim J. Jewett]

Who will benefit from this new warning?

Is this basically just changing builtins.open by adding:

if encoding is None and sys.flags.encoding_warning: # and not Android and 
not -X utf8 ?
warnings.warn(EncodingWarning("Are you sure you want locale instead of 
utf-8?"))

Even for the few people with the knowledge, time, interest, and authority to 
fix the code, is that really helpful?  

Helpful enough to put it directly in python as an optional mode, separate from 
the dev mode or show all warnings mode?  Why not just add it to a linter, or 
write a 2to3 style checker?  Or at least emit or not based on a warnings filter?

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GXZFCYFK7VOUSVZ5BVDCUW3JNJG6KPRS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Paul Moore]

On Thu, 11 Feb 2021 at 21:05, Jim J. Jewett  wrote:
>
> Who will benefit from this new warning?
>
> Is this basically just changing builtins.open by adding:
>
> if encoding is None and sys.flags.encoding_warning: # and not Android and 
> not -X utf8 ?
> warnings.warn(EncodingWarning("Are you sure you want locale instead 
> of utf-8?"))
>
> Even for the few people with the knowledge, time, interest, and authority to 
> fix the code, is that really helpful?
>
> Helpful enough to put it directly in python as an optional mode, separate 
> from the dev mode or show all warnings mode?  Why not just add it to a 
> linter, or write a 2to3 style checker?  Or at least emit or not based on a 
> warnings filter?

That's a very good point. If this warning is of use, why have none of
the well-known linters implemented it? And why not prototype the
proposal in them, at least? Python-ideas posts routinely get pushed to
justify "why can't this be done in an external library?" and that
probably applies here too.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VRBGH3ECNJBZMX7LINDNXYHQSKTKRTEX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

[Guido van Rossum]

It means "I can't prove it matches". This should be clear from the spec
already (it's an important point actually, since it means type checkers
cannot narrow in an else clause). So please don't file a PR to "add" this.

On Thu, Feb 11, 2021 at 11:49 AM Jim J. Jewett  wrote:

> If a TypeGuard returns false, does that mean "it doesn't match", or just
> "I can't prove it matches, but it still might"?  That seems relevant to the
> else clause ... and seems to have changed since the last version I looked
> at.
>
> -jJ
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KCJ2IIF4BUXGQEBLBPFNKPXZCGUHVIPD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5YP4NEEBV3NOHQZMDRTP5DV4DWWAYVFC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Inada Naoki]

On Fri, Feb 12, 2021 at 6:34 AM Paul Moore  wrote:
>
> On Thu, 11 Feb 2021 at 21:05, Jim J. Jewett  wrote:
> >
> > Who will benefit from this new warning?
> >
> > Is this basically just changing builtins.open by adding:
> >
> > if encoding is None and sys.flags.encoding_warning: # and not Android 
> > and not -X utf8 ?
> > warnings.warn(EncodingWarning("Are you sure you want locale instead 
> > of utf-8?"))
> >
> > Even for the few people with the knowledge, time, interest, and authority 
> > to fix the code, is that really helpful?
> >
> > Helpful enough to put it directly in python as an optional mode, separate 
> > from the dev mode or show all warnings mode?  Why not just add it to a 
> > linter, or write a 2to3 style checker?  Or at least emit or not based on a 
> > warnings filter?
>
> That's a very good point. If this warning is of use, why have none of
> the well-known linters implemented it? And why not prototype the
> proposal in them, at least? Python-ideas posts routinely get pushed to
> justify "why can't this be done in an external library?" and that
> probably applies here too.
>

* Linters can not add `encoding="locale"` to Python.
* This PEP provides the way to shift where warnings  is emitted.

def my_read_file(filename, encoding=None):
encoding = io.text_encoding(encoding)
with open(filename, encoding=encoding) with f:
return f.read()

This function is not warned. Caller of this function is warned
instead. It is difficult to implement in the Linter.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CKMRUBEI3UHEXSELZIQBA6NZCK77O75T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Inada Naoki]

On Fri, Feb 12, 2021 at 5:18 AM Jim J. Jewett  wrote:
>
> Inada Naoki wrote:
>
> > Default encoding is used for:
>
> > a. Really need to use locale specific encoding
> > b. UTF-8 (bug. not work on Windows)
> > c. ASCII (not a bug, but slow on Windows)
>
> > I assume most usages are (b) and (c). This PEP can reduce them soon.
>
> Is this just an assumption, based on those times being visible to someone who 
> installs a lot of packages, or has the use of any locale other than UTF-8 and 
> ASCII really gone down a lot?  Have browsers stopped using charset sniffing?
>

Using "most" is my fault. I am not good at Englsh. I should use "many" here.
You can see many bugs caused by not specifying `encoding="utf-8"` in Q&A sites.
I wrote some number about this common bugs in the PEP.

UTF-8 is used for 96.3% of web sites [1], although browser still use
charset sniffing. But how is it relating to this PEP?
[1] https://w3techs.com/technologies/details/en-utf8


> > Additionally, encoding="locale" will be backward/forward compatible
>
> What would be the problem with changing the default from None to locale?

It doesn't work on Python ~3.9.
So using `encoding="locale"` is not recommended anytime soon until
user drops Python 3.9 support.

> (I think you mentioned that they are the same 99% of the time; is that other 
> 1% likely to be cases where locale is wrong but None is right?  Would there 
> be a better way to represent that 1%?)
>

`encoding="locale"` and `encoding=None` has same behavior except
`encoding="locale"` doesn't emit EncodingWarning even when it is
opt-in.

There is little difference between `encoding=None` and
`encoding=locale.getpreferredencoding(False)`. The difference is:

* When Python is using Windows, and
* When when the file is console, and
* (for open()) When PYTHONLEGACYWINDOWSSTDIO is set
* (for TextIOWrapper()) When the file is not _WindowsConsoleIO

encoding=None uses console codepage but
encoding=locale.getpreferredencoding(False) uses
Otherwise, encoding=None and
encoding=locale.getpreferredencoding(False) are same.

So `encoding=locale.getpreferredencoding(False)` can be used to
specify locale-specific encoding explicitly.
But this PEP doesn't recommend it. This PEP recommend to use
EncodingWarning for just finding missing `encoding="utf-8"` (or any
other specific encoding).

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PD4BTBAQHFUYOCF5QKIBDIMHATPVEFPW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Inada Naoki]

On Thu, Feb 11, 2021 at 4:44 PM Jim J. Jewett  wrote:
>
> The PEP helps when the locale is ASCII or C, but that isn't enforced in 
> actual files.  I am confident that this is a frequent problem for packages 
> downloaded from mostly-English sites, including many software repositories.
>

The PEP helps developers living on UTF-8 locale to find missing
`encoding="utf-8"` bug.
This type of bug is very common, and many Windows users are suffered
by the bug when reading JSON, YAML, TOML, Markdown, or any other UTF-8
files.


> It does not seem to be a win when the locale is something incompatible with 
> utf-8, such as Latin-1, or whatever is still common in Japan.  The 
> surrogate-escape mechanism allows a proper round-trip, but python itself will 
> stop processing the characters correctly.
>

Surrogate-escape mechanism doesn't relating this PEP.


> For interactive use, when talking to another program (such as a terminal) 
> instead of an already existing file, the backwards compatibility problem 
> seems worse.
>

This PEP is 100% backward compatible.


> Changing the default to utf-8 (after a deprecation period showing how to make 
> locale an explicit default) may be reasonable, but claiming that it is 
> backwards compatible ... I didn't get that impression from the PEP.
>

This PEP doesn't propose to change the default encoding.

*If* we decide to change the default encoding in the future (maybe,
2025 or later) and start emitting DeprecationWarning where `encoding`
option is omitted, this PEP help it by:

* `encoding="locale"` option can be used since Python 3.10, and
* The number of DeprecationWarning shown is decreased because we can
add `encoding="utf-8"` many places before the time. At least, we can
fix all EncodingWarning in stdlib.

Maybe, the "Prepare to change the default encoding to UTF-8" is misleading.
I will try to fix the section or remove the section.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JBBRBR6AUTGP2SAVAUJVZJ3GM6FJQEBV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Eryk Sun]

On 2/11/21, Inada Naoki  wrote:
>
> There is little difference between `encoding=None` and
> `encoding=locale.getpreferredencoding(False)`. The difference is:
>
> * When Python is using Windows, and
> * When when the file is console, and
> * (for open()) When PYTHONLEGACYWINDOWSSTDIO is set
> * (for TextIOWrapper()) When the file is not _WindowsConsoleIO
>
> encoding=None uses console codepage but

os.device_encoding() -- i.e. _Py_device_encoding() -- only works for
hard-coded file descriptors 0, 1, and 2, instead of detecting a
console file. So opening "CON", "CONIN$", or "CONOUT$" has never used
the console input or output code page, nor has opening a duped
standard I/O fd such as open(os.dup(0)). It would be easy to
generalize _Py_device_encoding() to detect console files, but it's new
behavior.

Python 3.8+ introduced a bug (issue 42261) in which, even with legacy
standard I/O enabled and file descriptors 0-2, the console input and
output code pages are ignored. For example:

C:\>chcp 437
Active code page: 437
C:\>set PYTHONLEGACYWINDOWSSTDIO=1
C:\>py -3.9 -c "import sys; print(sys.stdout.encoding)"
cp1252

Regarding the last bullet point, io.TextIOWrapper doesn't know
anything about io._WindowsConsoleIO. The decision to use UTF-8 is in
io.open(). So manually wrapping a _WindowsConsoleIO file with
TextIOWrapper uses the locale preferred encoding instead of UTF-8. For
example:

>>> fb = open('conin$', 'rb')
>>> fb.raw
<_io._WindowsConsoleIO mode='rb' closefd=True>
>>> f = io.TextIOWrapper(fb)
>>> f.encoding
'cp1252'

I don't know whether it's worth making TextIOWrapper check for
_WindowsConsoleIO in order to make it use UTF-8. It's not common to
manually wrap a binary-mode file.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QBNH3XGSNBQ7XIJ5E542JIQ5Q5E63MCU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

[Sebastian Kreft]

I still think that we should reconsider deferring what happens to class and 
instance methods.

The arguments given in https://www.python.org/dev/peps/pep-0647/#id13 seem 
insufficient, specially considering than the workaround provided is quite 
awkward.

The author suggests to write def check(self, self2) -> Typeguard[T] and call it 
as self.check(self).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CA2MFS4XXDPC6PDBW53BZWLWEMQOBBP6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

[Guido van Rossum]

I think the use case (for x.is_foo()) is rare. And instead of writing
x.is_foo(x), if you make the guard a function you can write is_foo(x).

On Thu, Feb 11, 2021 at 6:51 PM Sebastian Kreft  wrote:

> I still think that we should reconsider deferring what happens to class
> and instance methods.
>
> The arguments given in https://www.python.org/dev/peps/pep-0647/#id13
> seem insufficient, specially considering than the workaround provided is
> quite awkward.
>
> The author suggests to write def check(self, self2) -> Typeguard[T] and
> call it as self.check(self).
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CA2MFS4XXDPC6PDBW53BZWLWEMQOBBP6/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MSGSGFJ6TBZMJBFZTQ5ORIAQH26TFZK3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Jim J. Jewett]

(I apologize if my summaries distort what Inada Naoki
 explained.)

He said that some people use the default None when they really want
either UTF-8 or ASCII.

My concern is that the warning will be a false alarm if they really do
need whatever locale returns, and that case may still be common.  (If
web browsers had stopped bothering to sniff for other charsets, then
maybe that situation really was getting rare.)

I asked when encoding=None is actually different from encoding=locale,
currently spelled encoding=locale.getpreferredencoding(False)

They can be different on Windows console, presumably because the
environment settings that control locale may differ from the charset
actually used by the console.  Even then, it only differs for open()
when PYTHONLEGACYWINDOWSSTDIO is set, and for TextIOWrapper() When the
file is not _WindowsConsoleIO

To me, that sounds narrow enough to be a windows issue, rather than an
issue with open.  Is there some way to write an encoding that sniffs
for charsets, particularly on windows, and to use that as the default
instead of assuming that locale will be correct?

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SSALFO3RTPX7QZ7B2MOWTZKYCJ5XKWK4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Jim J. Jewett]

On Thu, Feb 11, 2021 at 7:35 PM Inada Naoki  wrote:

> The PEP helps developers living on UTF-8 locale to find missing
> `encoding="utf-8"` bug.
> This type of bug is very common, and many Windows users are suffered
> by the bug when reading JSON, YAML, TOML, Markdown, or any other UTF-8
> files.

I think this is where we have been talking past each other.

You seem to be assuming that the programmer knows the correct
encoding, presumably because they (or their program) wrote it.  You
then assume that they neglected to mention the encoding out of
forgetfulness, perhaps because on their system, everything is always
UTF-8.  This clearly does happen, but the people who would make this
mistake most often -- they probably wouldn't think to test their code
under a special mode that catches only this.  (They might run a linter
that looked for all sorts of problems, including this.)

I instead assume that the programmer really doesn't know the encoding,
because the file is supplied by the user.  (The user may not know
either, since it is really supplied by some other program, but ...
neither python nor the programmer knows for sure.)  In this case, the
warning is not just a false alarm, but is actively misleading.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LHMFNFIOATO46NVOVCUOKFQCRWCZLY7M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Security releases of CPython

[Terry Reedy]

On 2/11/2021 3:23 PM, Michał Górny wrote:

Hello,

I'm the primary maintainer of CPython packages in Gentoo. I would like
to discuss possible improvement to the release process in order to
accelerate releasing security fixes to users.

I feel that vulnerability fixes do not make it to end users fast enough.


When we introduce a bad regression in a release, including a 
severe-enough security vulnerability, and discover it soon enough, we 
have sometimes made immediately releases.


Beyond that, your proposal to add several releases per Python version, 
perhaps double what we do now, runs up against the limits of volunteer 
time and effort.  As it is now, becoming a release manager is a 7 year 
commitment, with at least one release about every other month for the 
first 4.  There are also the 2 people who produce the Windows and macOS 
installers.  I believe each has done it for at least 7 or 8 years 
already.  Releases are not just a push of a button.  Make the release 
job too onerous, and there might not be any more volunteers.



For example, according to the current release schedules for 3.9 and 3.8,
the bugfix releases are planned two months apart. While the release is
expected to happen in the next few days, both versions are known to be
vulnerable for almost a month!

Ironically, things look even worse for security-supported versions.
Please correct me if I'm wrong but both 3.7 and 3.6 seem to be behind
schedule (planned for Jan 15th), and they are known to be vulnerable
since mid-October.

In my opinion, this causes three issues:

1. Users using official releases are exposed to security vulnerabilities
for prolonged periods of time.

2. When releases happen, security fixes are often combined with many
other changes. This causes problems for distribution maintainers who, on
one hand, would like to deploy the security fixes to production versions
ASAP, and on the other, would prefer that the new version remained in
testing for some time due to the other changes.

3. Effectively, it means that distribution developers need to track
and backport security fixes themselves. In the end, this means a lot of
duplicate work.


Perhaps people doing duplicate work could get together to eliminate some of
the duplication.  It should be possible to filter security commits from the
python-checkins list by looking at the news entries and if need be, the bpo
issues.


I think that security fixes are important enough to justify not sticking
to a strict release schedule. Therefore, I would like to propose that if
vulnerability fixes are committed, new releases are made
as frequently as necessary and as soon as possible (i.e. possibly giving
some time for testing) rather than according to a strict schedule.

Furthermore, I think that at least for branches that are in higher level
of maintenance than security, it could make sense to actually make
security releases (e.g. 3.9.1.x) that would include only security fixes
without other changes.


--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4S5LOTVJZUA5PQ5TRGQFCWHTGA4BOXBO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Inada Naoki]

On Fri, Feb 12, 2021 at 12:28 PM Jim J. Jewett  wrote:
>
> (I apologize if my summaries distort what Inada Naoki
>  explained.)
>
> He said that some people use the default None when they really want
> either UTF-8 or ASCII.

Yes. Even Python core developers.
For example: https://bugs.python.org/issue33684

This is just one example. I saw many codes using default encoding to
read JSON, YAML, TOML, Markdown, etc...


>
> My concern is that the warning will be a false alarm if they really do
> need whatever locale returns, and that case may still be common.  (If
> web browsers had stopped bothering to sniff for other charsets, then
> maybe that situation really was getting rare.)
>

That's one of reason why this warning is opt-in, like BytesWarning.

> I asked when encoding=None is actually different from encoding=locale,
> currently spelled encoding=locale.getpreferredencoding(False)
>

I don't understand this sentence. This PEP proposes
`encoding="locale"` that is equal to encoding=None but don't emit
EncodingWarning.

There was discussion about difference between `encoding=None` and
`encoding=locale.getpreferredencoding(False)` in this thread.


> They can be different on Windows console, presumably because the
> environment settings that control locale may differ from the charset
> actually used by the console.  Even then, it only differs for open()
> when PYTHONLEGACYWINDOWSSTDIO is set, and for TextIOWrapper() When the
> file is not _WindowsConsoleIO
>
> To me, that sounds narrow enough to be a windows issue, rather than an
> issue with open.

Yes. So if user want to specify locale-specific encoding and don't
want to drop Python 3.9 support, user can use
encoding=locale.getpreferredencoding(False).

But this PEP doesn't recommend it. Third party libraries can use
`encoding="locale"` after they drop Python 3.9 support.


>  Is there some way to write an encoding that sniffs
> for charsets, particularly on windows, and to use that as the default
> instead of assuming that locale will be correct?
>
> -jJ

There is no reliable way, AFAIK.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LJASRUN5G2PYEUOT7H34LGGBYEHBUB3C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 597: Add optional EncodingWarning

[Inada Naoki]

On Fri, Feb 12, 2021 at 12:45 PM Jim J. Jewett  wrote:
>
> On Thu, Feb 11, 2021 at 7:35 PM Inada Naoki  wrote:
>
> > The PEP helps developers living on UTF-8 locale to find missing
> > `encoding="utf-8"` bug.
> > This type of bug is very common, and many Windows users are suffered
> > by the bug when reading JSON, YAML, TOML, Markdown, or any other UTF-8
> > files.
>
> I think this is where we have been talking past each other.
>
> You seem to be assuming that the programmer knows the correct
> encoding, presumably because they (or their program) wrote it.

Not always, but many times.

>  You
> then assume that they neglected to mention the encoding out of
> forgetfulness, perhaps because on their system, everything is always
> UTF-8.  This clearly does happen, but the people who would make this
> mistake most often -- they probably wouldn't think to test their code
> under a special mode that catches only this.  (They might run a linter
> that looked for all sorts of problems, including this.)
>

Some Python experts can write `export PYTHONWARNENCODING=1` in their .bashrc.
They can find such mistakes not only in their codes but also in
libraries they are using.
Since they are experts, they can understand the warning and report it
to the library author correctly.

So this option helps library authors even if they don't use this option.


> I instead assume that the programmer really doesn't know the encoding,
> because the file is supplied by the user.  (The user may not know
> either, since it is really supplied by some other program, but ...
> neither python nor the programmer knows for sure.)
>  In this case, the
> warning is not just a false alarm, but is actively misleading.
>
> -jJ

This option is opt-in.  People don't understand what this warning
means should not opt-in the warning.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KLYUYKLHWCTTK7HOYNPDRPRS6WIQQU7K/
Code of Conduct: http://python.org/psf/codeofconduct/