[Python-Dev] Re: HighwayHash > SipHash?

[Victor Stinner]

Hi,

Le lun. 2 déc. 2019 à 08:18, Larry Hastings  a écrit :
> My random Googling turned up a new hash function tonight: "HighwayHash".  
> It's a keyed hash function like the SipHash we now use for hashing strings / 
> bytes / etc for our lovely dicts.
>
> Key points:
>
> Apache 2 license
> Can use SIMD
> "5x faster than SipHash"

It is designed to be more efficient thanks to SIMD.

> Since the choice of SipHash is a private implementation detail, and since we 
> all like fast things, is it worth considering switching to HighwayHash?  
> Don't ask me, I'm only a release manager (for a version nobody cares about 
> anymore).  These things are above my pay grade.

The question is more the number of cryptanalysis and if they are know weakness.

In the meanwhile, someone can maybe experiment a hack to use
HighwayHash in Python and run some microbenchmarks and pyperformance
benchmark suite.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SD3LFJQLJ7IBIGSW76V36E35LDWLVPPG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP proposal to limit various aspects of a Python program to one million.

[Mark Shannon]

Hi Everyone,

I am proposing a new PEP, still in draft form, to impose a limit of one 
million on various aspects of Python programs, such as the lines of code 
per module.


Any thoughts or feedback?

The PEP:
https://github.com/markshannon/peps/blob/one-million/pep-100.rst

Cheers,
Mark.


Full text
*

PEP: 100
Title: The one million limit
Author: Mark Shannon 
Status: Active
Type: Enhancement
Content-Type: text/x-rst
Created: 03-Dec-2019
Post-History:



Abstract

This PR proposes a limit of one million (1 000 000) for various aspects 
of Python code and its implementation.


The Python language does not specify limits for many of its features.
Not having any limit to these values seems to enhance programmer freedom,
at least superficially, but in practice the CPython VM and other Python 
virtual

machines have implicit limits or are forced to assume that the limits are
astronomical, which is expensive.

This PR lists a number of features which are to have a limit of one 
million.
If a language feature is not listed but appears unlimited and must be 
finite,
for physical reasons if no other, then a limit of one million should be 
assumed.



Motivation
==

There are many values that need to be represented in a virtual machine.
If no limit is specified for these values, then the representation must 
either be inefficient or vulnerable to overflow.

The CPython virtual machine represents values like line numbers,
stack offsets and instruction offsets by 32 bit values. This is 
inefficient, and potentially unsafe.


It is inefficient as actual values rarely need more than a dozen or so 
bits to represent them.


It is unsafe as malicious or poorly generated code could cause values to 
exceed 2\ :sup:`32`.


For example, line numbers are represented by 32 bit values internally.
This is inefficient, given that modules almost never exceed a few 
thousand lines.

Despite being inefficent, is is still vulnerable to overflow as
it is easy for an attacker to created a module with billions of newline 
characters.


Memory access is usually a limiting factor in the performance of modern 
CPUs.
Better packing of data structures enhances locality and reduces memory 
bandwith,

at a modest increase in ALU usage (for shifting and masking).
Being able to safely store important values in 20 bits would allow 
memory savings

in several data structures including, but not limited to:

* Frame objects
* Object headers
* Code objects

There is also the potential for a more efficient instruction format, 
speeding up interpreter dispatch.


Rationale
=

Imposing a limit on values such as lines of code in a module, and the 
number of local variables,
has significant advantages for ease of implementation and efficiency of 
virtual machines.
If the limit is sufficiently large, there is no adverse effect on users 
of the language.


By selecting a fixed but large limit for these values,
it is possible to have both safety and efficiency whilst causing no 
inconvience to human programmers

and only very rare problems for code generators.

One million
---

The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1 
(65535) for many program

elements similar to those covered here.
This limit enables limited values to fit in 16 bits, which is a very 
efficient machine representation.
However, this limit is quite easily exceeded in practice by code 
generators and
the author is aware of existing Python code that already exceeds 2\ 
:sup:`16` lines of code.


A limit of one million fits into 20 bits which, although not as 
convenient for machine representation,
is still reasonably compact. Three signed valuses in the range -1000_000 
to +1000_000 can fit into a 64 bit word.
A limit of one million is small enough for efficiency advantages (only 
20 bits),
but large enough not to impact users (no one has ever written a module 
of one million lines).


The value "one million" is very easy to remember.

Isn't this "640K ought to be enough for anybody" again?
---

The infamous 640K memory limit was a limit on machine usable resources.
The proposed one million limit is a limit on human generated code.

While it is possible that generated code could exceed the limit,
it is easy for a code generator to modify its output to conform.
The author has hit the 64K limit in the JVM on at least two occasions 
when generating Java code.

The workarounds were relatively straightforward and
probably wouldn't have been necessary with a limit of one million 
bytecodes or lines of code.



Specification
=

This PR proposes that the following language features and runtime values 
be limited to one million.


* The number of source code lines in a module
* The number of bytecode instructions in a code object.
* The sum of local variables and stack usage for a code object.
* The number of distinct names in a code object
* The number of con

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Paddy McCarthy]

I would not have such a small limit.
I can envisage generating code from a log then evaluating that code. 1
million lines could be small, given the speed of the interpreter on modern
machines.
One might want to generate data as a Python file rather than a pile and
load that as a module. There might well be alternate methods, but if it
works and is quickly debugged at small scale, then you put a possible extra
barrier in the way.
Python is still a scripting language, why limit a "quick hack".

On Tue, Dec 3, 2019, 4:22 PM Mark Shannon  wrote:

> Hi Everyone,
>
> I am proposing a new PEP, still in draft form, to impose a limit of one
> million on various aspects of Python programs, such as the lines of code
> per module.
>
> Any thoughts or feedback?
>
> The PEP:
> https://github.com/markshannon/peps/blob/one-million/pep-100.rst
>
> Cheers,
> Mark.
>
>
> Full text
> *
>
> PEP: 100
> Title: The one million limit
> Author: Mark Shannon 
> Status: Active
> Type: Enhancement
> Content-Type: text/x-rst
> Created: 03-Dec-2019
> Post-History:
>
>
>
> Abstract
> 
> This PR proposes a limit of one million (1 000 000) for various aspects
> of Python code and its implementation.
>
> The Python language does not specify limits for many of its features.
> Not having any limit to these values seems to enhance programmer freedom,
> at least superficially, but in practice the CPython VM and other Python
> virtual
> machines have implicit limits or are forced to assume that the limits are
> astronomical, which is expensive.
>
> This PR lists a number of features which are to have a limit of one
> million.
> If a language feature is not listed but appears unlimited and must be
> finite,
> for physical reasons if no other, then a limit of one million should be
> assumed.
>
>
> Motivation
> ==
>
> There are many values that need to be represented in a virtual machine.
> If no limit is specified for these values, then the representation must
> either be inefficient or vulnerable to overflow.
> The CPython virtual machine represents values like line numbers,
> stack offsets and instruction offsets by 32 bit values. This is
> inefficient, and potentially unsafe.
>
> It is inefficient as actual values rarely need more than a dozen or so
> bits to represent them.
>
> It is unsafe as malicious or poorly generated code could cause values to
> exceed 2\ :sup:`32`.
>
> For example, line numbers are represented by 32 bit values internally.
> This is inefficient, given that modules almost never exceed a few
> thousand lines.
> Despite being inefficent, is is still vulnerable to overflow as
> it is easy for an attacker to created a module with billions of newline
> characters.
>
> Memory access is usually a limiting factor in the performance of modern
> CPUs.
> Better packing of data structures enhances locality and reduces memory
> bandwith,
> at a modest increase in ALU usage (for shifting and masking).
> Being able to safely store important values in 20 bits would allow
> memory savings
> in several data structures including, but not limited to:
>
> * Frame objects
> * Object headers
> * Code objects
>
> There is also the potential for a more efficient instruction format,
> speeding up interpreter dispatch.
>
> Rationale
> =
>
> Imposing a limit on values such as lines of code in a module, and the
> number of local variables,
> has significant advantages for ease of implementation and efficiency of
> virtual machines.
> If the limit is sufficiently large, there is no adverse effect on users
> of the language.
>
> By selecting a fixed but large limit for these values,
> it is possible to have both safety and efficiency whilst causing no
> inconvience to human programmers
> and only very rare problems for code generators.
>
> One million
> ---
>
> The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
> (65535) for many program
> elements similar to those covered here.
> This limit enables limited values to fit in 16 bits, which is a very
> efficient machine representation.
> However, this limit is quite easily exceeded in practice by code
> generators and
> the author is aware of existing Python code that already exceeds 2\
> :sup:`16` lines of code.
>
> A limit of one million fits into 20 bits which, although not as
> convenient for machine representation,
> is still reasonably compact. Three signed valuses in the range -1000_000
> to +1000_000 can fit into a 64 bit word.
> A limit of one million is small enough for efficiency advantages (only
> 20 bits),
> but large enough not to impact users (no one has ever written a module
> of one million lines).
>
> The value "one million" is very easy to remember.
>
> Isn't this "640K ought to be enough for anybody" again?
> ---
>
> The infamous 640K memory limit was a limit on machine usable resources.
> The proposed one million limit is a limit on human generated code.
>
> While 

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Chris Angelico]

On Wed, Dec 4, 2019 at 3:20 AM Mark Shannon  wrote:
>
> Hi Everyone,
>
> I am proposing a new PEP, still in draft form, to impose a limit of one
> million on various aspects of Python programs, such as the lines of code
> per module.
>
> Any thoughts or feedback?
>

The main justification for these is performance, if I'm reading this
correctly. Have you measured anything to find out just how much you
gain? Arbitrary limits are always annoying when you hit them, so it
would be nice to see how much benefit there is first.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MD3IOMWQCGMR3JDZGYZO2JE4P2Z5XIMZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Deprecating the "u" string literal prefix

[Serhiy Storchaka]

The 'u" string literal prefix was removed in 3.0 and reintroduced in 3.3 
to help writing the code compatible with Python 2 and 3 [1]. After the 
dead of Python 2.7 we will remove some deprecated features kept for 
compatibility with 2.7. When we are going to deprecate and remove the 
"u" prefix?


[1] https://www.python.org/dev/peps/pep-0414/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EMFDKQ57JVWUZ6TPZM5VTFW7EUKVYAOY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Steve Dower]

On 03Dec2019 0815, Mark Shannon wrote:

Hi Everyone,

I am proposing a new PEP, still in draft form, to impose a limit of one 
million on various aspects of Python programs, such as the lines of code 
per module.


I assume you're aiming for acceptance in just under four months? :)


Any thoughts or feedback?


It's actually not an unreasonable idea, to be fair. Picking an arbitrary 
limit less than 2**32 is certainly safer for many reasons, and very 
unlikely to impact real usage. We already have some real limits well 
below 10**6 (such as if/else depth and recursion limits).


That said, I don't really want to impact edge-case usage, and I'm all 
too familiar with other examples of arbitrary limits (no file system 
would need a path longer than 260 characters, right? :o) ).


Some comments on the specific items, assuming we're not just going to 
reject this out of hand.



Specification
=

This PR proposes that the following language features and runtime values 
be limited to one million.


* The number of source code lines in a module


This one feels the most arbitrary. What if I have a million blank lines 
or comments? We still need the correct line number to be stored, which 
means our lineno fields still have to go beyond 10**6. Limiting total 
lines in a module to 10**6 is certainly too small.



* The number of bytecode instructions in a code object.


Seems reasonable.


* The sum of local variables and stack usage for a code object.


I suspect our effective limit is already lower than 10**6 here anyway - 
do we know what it actually is?



* The number of distinct names in a code object


SGTM.


* The number of constants in a code object.


SGTM.


* The number of classes in a running interpreter.


I'm a little hesitant on this one, but perhaps there's a way to use a 
sentinel for class_id (in your later struct) for when someone exceeds 
this limit? The benefits seem worthwhile here even without the rest of 
the PEP.



* The number of live coroutines in a running interpreter.


SGTM. At this point we're probably putting serious pressure on kernel 
wait objects/FDs anyway, and if you're not waiting then you're probably 
not efficiently using coroutines anyway.



Having 20 bit operands (21 bits for relative branches) allows instructions
to fit into 32 bits without needing additional ``EXTENDED_ARG`` 
instructions.
This improves dispatch, as the operand is strictly local to the 
instruction.

Using super-instructions would make that the 32 bit format
almost as compact as the 16 bit format, and significantly faster.


We can measure this - how common are EXTENDED_ARG instructions? ISTR we 
checked this when switching to 16-bit instructions and it was worth it, 
but I'm not sure whether we also considered 32-bit instructions at that 
time.



Total number of classes in a running interpreter


This limit has to the potential to reduce the size of object headers 
considerably.


This would be awesome, and I *think* it's ABI compatible (as the 
affected fields are all put behind the PyObject* that gets returned, 
right?). If so, I think it's worth calling that out in the text, as it's 
not immediately obvious.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OQ4N23YAVNOYZKHZEPBO37PJFO4XX7HP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Guido van Rossum]

I think it’s too soon to worry about this. I don’t see a reason to harass
people who maintain code based that were just recently migrated.

On Tue, Dec 3, 2019 at 09:21 Serhiy Storchaka  wrote:

> The 'u" string literal prefix was removed in 3.0 and reintroduced in 3.3
> to help writing the code compatible with Python 2 and 3 [1]. After the
> dead of Python 2.7 we will remove some deprecated features kept for
> compatibility with 2.7. When we are going to deprecate and remove the
> "u" prefix?
>
> [1] https://www.python.org/dev/peps/pep-0414/
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EMFDKQ57JVWUZ6TPZM5VTFW7EUKVYAOY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DWNANYD2PYTSOOQBOCSNSIZTHSUUTQED/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Rhodri James]

On 03/12/2019 17:16, Serhiy Storchaka wrote:
The 'u" string literal prefix was removed in 3.0 and reintroduced in 3.3 
to help writing the code compatible with Python 2 and 3 [1]. After the 
dead of Python 2.7 we will remove some deprecated features kept for 
compatibility with 2.7. When we are going to deprecate and remove the 
"u" prefix?


[1] https://www.python.org/dev/peps/pep-0414/


You are presupposing that the answer to the question "Are we going to 
deprecate and remove the "u" prefix" is "Yes".  I'm not convinced it is.


--
Rhodri James *-* Kynesim Ltd
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YOCNBXZ7K3VN4UBF7ASBKPWB32QAG63L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Rhodri James]

On 03/12/2019 16:15, Mark Shannon wrote:

Hi Everyone,

I am proposing a new PEP, still in draft form, to impose a limit of one 
million on various aspects of Python programs, such as the lines of code 
per module.


Any thoughts or feedback?

The PEP:
https://github.com/markshannon/peps/blob/one-million/pep-100.rst

Cheers,
Mark.


Full text
*

PEP: 100
Title: The one million limit
Author: Mark Shannon 
Status: Active
Type: Enhancement
Content-Type: text/x-rst
Created: 03-Dec-2019
Post-History:



Abstract

This PR proposes a limit of one million (1 000 000) for various aspects 
of Python code and its implementation.


The Python language does not specify limits for many of its features.
Not having any limit to these values seems to enhance programmer freedom,
at least superficially, but in practice the CPython VM and other Python 
virtual

machines have implicit limits or are forced to assume that the limits are
astronomical, which is expensive.

This PR lists a number of features which are to have a limit of one 
million.
If a language feature is not listed but appears unlimited and must be 
finite,
for physical reasons if no other, then a limit of one million should be 
assumed.



Motivation
==

There are many values that need to be represented in a virtual machine.
If no limit is specified for these values, then the representation must 
either be inefficient or vulnerable to overflow.

The CPython virtual machine represents values like line numbers,
stack offsets and instruction offsets by 32 bit values. This is 
inefficient, and potentially unsafe.


It is inefficient as actual values rarely need more than a dozen or so 
bits to represent them.


It is unsafe as malicious or poorly generated code could cause values to 
exceed 2\ :sup:`32`.


For example, line numbers are represented by 32 bit values internally.
This is inefficient, given that modules almost never exceed a few 
thousand lines.

Despite being inefficent, is is still vulnerable to overflow as
it is easy for an attacker to created a module with billions of newline 
characters.


Memory access is usually a limiting factor in the performance of modern 
CPUs.
Better packing of data structures enhances locality and reduces memory 
bandwith,

at a modest increase in ALU usage (for shifting and masking).
Being able to safely store important values in 20 bits would allow 
memory savings

in several data structures including, but not limited to:

* Frame objects
* Object headers
* Code objects

There is also the potential for a more efficient instruction format, 
speeding up interpreter dispatch.


Rationale
=

Imposing a limit on values such as lines of code in a module, and the 
number of local variables,
has significant advantages for ease of implementation and efficiency of 
virtual machines.
If the limit is sufficiently large, there is no adverse effect on users 
of the language.


By selecting a fixed but large limit for these values,
it is possible to have both safety and efficiency whilst causing no 
inconvience to human programmers

and only very rare problems for code generators.

One million
---

The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1 
(65535) for many program

elements similar to those covered here.
This limit enables limited values to fit in 16 bits, which is a very 
efficient machine representation.
However, this limit is quite easily exceeded in practice by code 
generators and
the author is aware of existing Python code that already exceeds 2\ 
:sup:`16` lines of code.


A limit of one million fits into 20 bits which, although not as 
convenient for machine representation,
is still reasonably compact. Three signed valuses in the range -1000_000 
to +1000_000 can fit into a 64 bit word.
A limit of one million is small enough for efficiency advantages (only 
20 bits),
but large enough not to impact users (no one has ever written a module 
of one million lines).


OK, let me stop you here.  If you have twenty bits of information, 
you'll be fitting them into a 32-bit word anyway.  Anything else will be 
more or less inefficient to access, depending on your processor.  You 
aren't going to save anything there.


If you have plans to use the spare bits for something else, please 
don't.  I've seen this done in two major architectures (status flags for 
both the IBM System/370 and ARM 2 and 3 architectures lived in the top 
bits of the program counter), and it was acknowledged to be a major 
mistake both times.  Aside from limiting your expansion (Who would ever 
want more than 24 bits of address space?  Everyone, it turns out :-), 
every access you make to that word is going to need to mask out some 
bits of the word.  You would take an efficiency hit on every access.




Isn't this "640K ought to be enough for anybody" again?
---

The infamous 640K memory limit was a limit on machine usable resources.

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Barry Warsaw]

On Dec 3, 2019, at 09:16, Serhiy Storchaka  wrote:
> 
> The 'u" string literal prefix was removed in 3.0 and reintroduced in 3.3 to 
> help writing the code compatible with Python 2 and 3 [1]. After the dead of 
> Python 2.7 we will remove some deprecated features kept for compatibility 
> with 2.7. When we are going to deprecate and remove the "u" prefix?

I think removing or even deprecating it will cause unnecessary code churn in 
third party libraries, for very little value.  Let’s just call it a wart until 
Python 4000.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JBOHEFZ6PXSIKBBLNILGKIFYWQZ2WAHZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Ethan Furman]

On 12/03/2019 09:16 AM, Serhiy Storchaka wrote:


The 'u" string literal prefix was removed in 3.0 and reintroduced in 3.3 to help writing 
the code compatible with Python 2 and 3 [1]. After the dead of Python 2.7 we will remove some 
deprecated features kept for compatibility with 2.7. When we are going to deprecate and remove 
the "u" prefix?


Can we gather all the 2.7 compatibility shims and remove them in 3.9?  If not, I would 
say let's deprecate whatever needs deprecating and remove them in 3.10.  I don't think 
the u'' prefix needs the normal 2-cyle deprecation since it was added specifically for 
2/3 cross-version code, and, as you say, "the dead of Python 2.7" has occurred 
(at least for us).

--
~Ethan~

 

[1] https://www.python.org/dev/peps/pep-0414/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GTCA6FGKF4IVCKW5HFTAP42LC4VPGLLR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[francismb]

On 12/3/19 6:31 PM, Guido van Rossum wrote:
> I think it’s too soon to worry about this. I don’t see a reason to harass
> people who maintain code based that were just recently migrated.
+1  ... or code that will be migrated in the (near) future ;-) ...


Regards,
francismb
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZCQKLLBGCKZRPIA2RGUQHOO5JRP6LEJW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Christian Heimes]

On 03/12/2019 19.09, Barry Warsaw wrote:
> On Dec 3, 2019, at 09:16, Serhiy Storchaka 
> wrote:
>>
>> The 'u" string literal prefix was removed in 3.0 and reintroduced
>> in 3.3 to help writing the code compatible with Python 2 and 3
>> [1]. After the dead of Python 2.7 we will remove some deprecated
>> features kept for compatibility with 2.7. When we are going to
>> deprecate and remove the "u" prefix?
>
> I think removing or even deprecating it will cause unnecessary code
> churn in third party libraries, for very little value.  Let’s just
> call it a wart until Python 4000.

I'm 100% with Barry. We can certainly document the "u" string prefix
as deprecated. I'm strongly against removing it from Python 3 or even
raising a deprecation warning. Let's leave it to linters and maybe
consider deprecation for Python 4. Or maybe not.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4GGYZ6XNEH5OQDYC6VLH2MADUVD5LGF3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Ethan Furman]

On 12/03/2019 09:31 AM, Guido van Rossum wrote:


I think it’s too soon to worry about this. I don’t see a reason to harass 
people who maintain code based that were just recently migrated.


I'm happy to go with this, since my libraries still do the 2/3 straddle.

Do we want to set a date/version where we will drop code that only exists to 
make 2/3 projects easier?

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B7ZK6TQ2L6667R54FGUSGW3H2ELJ243F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Simon Cross]

On Tue, Dec 3, 2019 at 7:42 PM Guido van Rossum  wrote:

> I think it’s too soon to worry about this. I don’t see a reason to harass
> people who maintain code based that were just recently migrated.
>

+100
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PSPKIXP4LE6UJBIJAVFECHVWK7GSI5KF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Christian Heimes]

On 03/12/2019 21.04, Ethan Furman wrote:
> On 12/03/2019 09:31 AM, Guido van Rossum wrote:
> 
>> I think it’s too soon to worry about this. I don’t see a reason to
>> harass people who maintain code based that were just recently migrated.
> 
> I'm happy to go with this, since my libraries still do the 2/3 straddle.
> 
> Do we want to set a date/version where we will drop code that only
> exists to make 2/3 projects easier?

How about when the last major Linux distro with Python 2 reaches end of
general support? According to Wikipedia that would be 2023 for Ubuntu
18.04 LTS, 2024 for RHEL 8.0, and 2028 for SUSE SLES 15.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3XZZ7FSE3W57VSFBHCNAXIEKTLQWPJBD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Skip Montanaro]

Guido> I think it’s too soon to worry about this.
Simon> +100

Ditto. Besides, isn't support for u"..." just a variable and a couple
tests in the earliest phase of compilation? If things are going to get
deprecated/removed, I'd prefer the focus be placed on those bits which
present a significant support burden.

Skip
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WFB7KNWCFK3U2HD6I5UCUT6RQFCB7BMJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Antoine Pitrou]

On Tue, 3 Dec 2019 09:31:22 -0800
Guido van Rossum  wrote:
> I think it’s too soon to worry about this. I don’t see a reason to harass
> people who maintain code based that were just recently migrated.

Agreed with Guido.  Let's wait a couple more years and rethink it.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IMQQLXZYUFSR43I7KM6V7K2KEQQFIYCY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Eric V. Smith]

On 12/3/2019 3:35 PM, Christian Heimes wrote:

On 03/12/2019 21.04, Ethan Furman wrote:

On 12/03/2019 09:31 AM, Guido van Rossum wrote:


I think it’s too soon to worry about this. I don’t see a reason to
harass people who maintain code based that were just recently migrated.

I'm happy to go with this, since my libraries still do the 2/3 straddle.

Do we want to set a date/version where we will drop code that only
exists to make 2/3 projects easier?

How about when the last major Linux distro with Python 2 reaches end of
general support? According to Wikipedia that would be 2023 for Ubuntu
18.04 LTS, 2024 for RHEL 8.0, and 2028 for SUSE SLES 15.


I agree we should wait a long time before removing the 'u' prefix. If 
I'm using a library that uses 'u', but that library is no longer 
maintained, when do we want to break my app, just because I use this 
library? Given that 'u' requires so little code to implement, I don't 
see what's to be gained by removing it. Maybe document it as deprecated 
(and why), but that's about it.


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/44QX46DVMO7MDXKNPXXLEIADNUTR7ZMS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Multiple reference leaks in master

[Pablo Galindo Salgado]

Hi,

Me (and Victor) have not been able to attend the buildbots for a while
unfortunately and today
I was checking them out after some fixes to the SSL tests and sadly the
refleaks buildbots have many
independent issues. At least these tests are failing due to reference
leaks:

test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
test_threading

I am trying to slowly fix them but finding them and debugging often takes
several hours per leak. I was tracking them
in https://bugs.python.org/issue38962 until I discover that there are
multiple sources of leaks so I may open more issues.

Please, when you merge a PR, take a look at the buildbots (especially the
refleak buildbots that do not report still to the PR
in case of failure) so the issues don't keep piling up, as multiple
refleaks together are much difficult to deal with.

Thanks, everyone!

Regards from cloudy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Ned Batchelder]

On 12/3/19 2:41 PM, Christian Heimes wrote:

On 03/12/2019 19.09, Barry Warsaw wrote:

On Dec 3, 2019, at 09:16, Serhiy Storchaka 
wrote:

The 'u" string literal prefix was removed in 3.0 and reintroduced
in 3.3 to help writing the code compatible with Python 2 and 3
[1]. After the dead of Python 2.7 we will remove some deprecated
features kept for compatibility with 2.7. When we are going to
deprecate and remove the "u" prefix?

I think removing or even deprecating it will cause unnecessary code
churn in third party libraries, for very little value.  Let’s just
call it a wart until Python 4000.

I'm 100% with Barry. We can certainly document the "u" string prefix
as deprecated. I'm strongly against removing it from Python 3 or even
raising a deprecation warning. Let's leave it to linters and maybe
consider deprecation for Python 4. Or maybe not.


Yes, please, please, don't remove this prefix. It will cause needless 
churn, and force people to revisit code that works just fine.  We should 
try to keep old code working unless there is a clear benefit in making a 
breaking change. I haven't heard a benefit to removing this prefix.


"We added it just to help people migrate" is not a reason to remove it now.

--Ned.



Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4GGYZ6XNEH5OQDYC6VLH2MADUVD5LGF3/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2DBWZFPYW7XOGMO3DT6UEBQMLMVJQMQ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Ned Batchelder]

On 12/3/19 12:16 PM, Serhiy Storchaka wrote:
The 'u" string literal prefix was removed in 3.0 and reintroduced in 
3.3 to help writing the code compatible with Python 2 and 3 [1]. After 
the dead of Python 2.7 we will remove some deprecated features kept 
for compatibility with 2.7. When we are going to deprecate and remove 
the "u" prefix?


[1] https://www.python.org/dev/peps/pep-0414/


Can you clarify this?: "we will remove some deprecated features kept for 
compatibility with 2.7"  Where is the discussion of the features being 
removed?


--Ned.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PY24FGY3GLHZP6X4G2UPYEAD5INVEHHV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

[Guido van Rossum]

How do I find the refleak buildbots? I went to the devguide and searched
for "buildbot", which pointed to https://www.python.org/dev/buildbot/ --
but searching there for "refleak" finds nothing.

On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado 
wrote:

> Hi,
>
> Me (and Victor) have not been able to attend the buildbots for a while
> unfortunately and today
> I was checking them out after some fixes to the SSL tests and sadly the
> refleaks buildbots have many
> independent issues. At least these tests are failing due to reference
> leaks:
>
> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
> test_threading
>
> I am trying to slowly fix them but finding them and debugging often takes
> several hours per leak. I was tracking them
> in https://bugs.python.org/issue38962 until I discover that there are
> multiple sources of leaks so I may open more issues.
>
> Please, when you merge a PR, take a look at the buildbots (especially the
> refleak buildbots that do not report still to the PR
> in case of failure) so the issues don't keep piling up, as multiple
> refleaks together are much difficult to deal with.
>
> Thanks, everyone!
>
> Regards from cloudy London,
> Pablo Galindo Salgado
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/23VHKPOXDRI2CD52VCE2YU3CTTIHGIDO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

[Pablo Galindo Salgado]

> How do I find the refleak buildbots?

In this page:

https://buildbot.python.org/all/#/builders

all the buildbots that have "Refleaks" in the name. You can click on the
tags to filter only the "stable" and the ones in master "3.x".

On Tue, 3 Dec 2019 at 22:11, Guido van Rossum  wrote:

> How do I find the refleak buildbots? I went to the devguide and searched
> for "buildbot", which pointed to https://www.python.org/dev/buildbot/ --
> but searching there for "refleak" finds nothing.
>
> On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado 
> wrote:
>
>> Hi,
>>
>> Me (and Victor) have not been able to attend the buildbots for a while
>> unfortunately and today
>> I was checking them out after some fixes to the SSL tests and sadly the
>> refleaks buildbots have many
>> independent issues. At least these tests are failing due to reference
>> leaks:
>>
>> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
>> test_threading
>>
>> I am trying to slowly fix them but finding them and debugging often takes
>> several hours per leak. I was tracking them
>> in https://bugs.python.org/issue38962 until I discover that there are
>> multiple sources of leaks so I may open more issues.
>>
>> Please, when you merge a PR, take a look at the buildbots (especially the
>> refleak buildbots that do not report still to the PR
>> in case of failure) so the issues don't keep piling up, as multiple
>> refleaks together are much difficult to deal with.
>>
>> Thanks, everyone!
>>
>> Regards from cloudy London,
>> Pablo Galindo Salgado
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4EB7QF7VV4LRJ4XRHYERJI5327OHGYHW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Random832]

On Tue, Dec 3, 2019, at 12:22, Steve Dower wrote:
> > * The number of constants in a code object.
> 
> SGTM.

Two things...

First, for this one in particular, the number of constants in a code object is 
hard to predict. For example, recently (I want to say 3.7), the number of 
constants generated for most code was reduced by removing duplicate constants 
and intermediate constants that are optimized away by constant-folding. Adding 
the "x in [list or set literal]" optimization introduces a tuple or frozenset 
constant (and removes the constants for the items)

You also mentioned names - it's easy to imagine an implementation whose version 
of a code object shares names and constants in a single array with a single 
limit, or which has constants split up per type [and maybe doesn't implement 
tuple and frozenset constants at all] and shares names and string constants. Or 
one which does not store names at all for local variables in optimized code.

I also don't think characteristics of cpython-specific structures like code 
objects should be where limits are defined. One of the purposes of specifying a 
formal limit is to give other implementations a clear target for minimum 
support. In C, there's a long list of limits like these [some of which are 
quite a bit smaller than what typical implementations support], but all of them 
are for characteristics that can actually be determined by looking at the 
source code for a program.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/67OFA4G6A2X3XRBLG6XR4PWLEYBJLNLV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Last call for comments on PEP 573 (Module State Access from C Extension Methods)

[Nick Coghlan]

I have a few minor copy-editing comments, but I'll submit those as a PR to the 
PEPs repo (it's nothing substantial, just a few wording clarifications, and 
making sure the list of added methods is complete).

Thanks to you and everyone else for the work on this!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T2YHD5OWM4FQGRSX7DUAYMPRFCKNVGFT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Greg Ewing]

On 4/12/19 8:41 am, Christian Heimes wrote:

I'm strongly against removing it from Python 3 or even
raising a deprecation warning.


I agree. I know there is a maintenance cost to keeping
things like this around, but in this case it's pretty
minor. We've probably already spent more time discussing
it than it will cost to keep it for the rest of Python's
lifetime.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OTYLQJEJDPKU64QQ6F4B6HH3FPIHRVQE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

[Guido van Rossum]

Maybe you or Victor can update the devguide with this information? This is
not the first time that we've been asked to pay attention to the refleaks
buildbots, and it's frustrating that they are so hard to find.

On Tue, Dec 3, 2019 at 2:17 PM Pablo Galindo Salgado 
wrote:

> > How do I find the refleak buildbots?
>
> In this page:
>
> https://buildbot.python.org/all/#/builders
>
> all the buildbots that have "Refleaks" in the name. You can click on the
> tags to filter only the "stable" and the ones in master "3.x".
>
> On Tue, 3 Dec 2019 at 22:11, Guido van Rossum  wrote:
>
>> How do I find the refleak buildbots? I went to the devguide and searched
>> for "buildbot", which pointed to https://www.python.org/dev/buildbot/ --
>> but searching there for "refleak" finds nothing.
>>
>> On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado 
>> wrote:
>>
>>> Hi,
>>>
>>> Me (and Victor) have not been able to attend the buildbots for a while
>>> unfortunately and today
>>> I was checking them out after some fixes to the SSL tests and sadly the
>>> refleaks buildbots have many
>>> independent issues. At least these tests are failing due to reference
>>> leaks:
>>>
>>> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
>>> test_threading
>>>
>>> I am trying to slowly fix them but finding them and debugging often
>>> takes several hours per leak. I was tracking them
>>> in https://bugs.python.org/issue38962 until I discover that there are
>>> multiple sources of leaks so I may open more issues.
>>>
>>> Please, when you merge a PR, take a look at the buildbots (especially
>>> the refleak buildbots that do not report still to the PR
>>> in case of failure) so the issues don't keep piling up, as multiple
>>> refleaks together are much difficult to deal with.
>>>
>>> Thanks, everyone!
>>>
>>> Regards from cloudy London,
>>> Pablo Galindo Salgado
>>>
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him **(why is my pronoun here?)*
>> 
>>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UHSOYEXKJXLEZGGU7QZ5DS5Z7OM4S6FV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Rejecting PEP 606 and 608

[Barry Warsaw]

At the Steering Council’s November 12th meeting, we unanimously agreed to 
reject PEPs 606 and 608:

* 606 - Python Compatibility Version
* 608 - Coordinated Python Release

It was our opinion that neither PEP will effectively improve compatibility as 
proposed.  Additionally, PEP 606 had the problem of using global state to 
request compatibility, and PEP 608 puts external projects in the critical path 
for Python releases.

We want to thank Victor Stinner and Miro Hrončok for their contributions and 
authorship of these PEPs.  I apologize for being remiss in not sending this 
email before now.  I will shortly be submitting and merging a PR to official 
reject both PEPs.

-Barry, on behalf of the Python Steering Council



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6IVVUGDJZW7CLTXEXZFLQVHC3GG6FWKM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

[Pablo Galindo Salgado]

> Maybe you or Victor can update the devguide with this information? This
is not the first time that we've been asked to pay attention to the
refleaks buildbots, and it's frustrating that they are so hard to find.

Good point! I will make a PR to the dev guide improving the existing
documentation and I will think of other ways to make discovery easier.

On Wed, 4 Dec 2019 at 00:00, Guido van Rossum  wrote:

> Maybe you or Victor can update the devguide with this information? This is
> not the first time that we've been asked to pay attention to the refleaks
> buildbots, and it's frustrating that they are so hard to find.
>
> On Tue, Dec 3, 2019 at 2:17 PM Pablo Galindo Salgado 
> wrote:
>
>> > How do I find the refleak buildbots?
>>
>> In this page:
>>
>> https://buildbot.python.org/all/#/builders
>>
>> all the buildbots that have "Refleaks" in the name. You can click on the
>> tags to filter only the "stable" and the ones in master "3.x".
>>
>> On Tue, 3 Dec 2019 at 22:11, Guido van Rossum  wrote:
>>
>>> How do I find the refleak buildbots? I went to the devguide and searched
>>> for "buildbot", which pointed to https://www.python.org/dev/buildbot/
>>> -- but searching there for "refleak" finds nothing.
>>>
>>> On Tue, Dec 3, 2019 at 1:16 PM Pablo Galindo Salgado <
>>> pablog...@gmail.com> wrote:
>>>
 Hi,

 Me (and Victor) have not been able to attend the buildbots for a while
 unfortunately and today
 I was checking them out after some fixes to the SSL tests and sadly the
 refleaks buildbots have many
 independent issues. At least these tests are failing due to reference
 leaks:

 test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
 test_threading

 I am trying to slowly fix them but finding them and debugging often
 takes several hours per leak. I was tracking them
 in https://bugs.python.org/issue38962 until I discover that there are
 multiple sources of leaks so I may open more issues.

 Please, when you merge a PR, take a look at the buildbots (especially
 the refleak buildbots that do not report still to the PR
 in case of failure) so the issues don't keep piling up, as multiple
 refleaks together are much difficult to deal with.

 Thanks, everyone!

 Regards from cloudy London,
 Pablo Galindo Salgado

 ___
 Python-Dev mailing list -- python-dev@python.org
 To unsubscribe send an email to python-dev-le...@python.org
 https://mail.python.org/mailman3/lists/python-dev.python.org/
 Message archived at
 https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
 Code of Conduct: http://python.org/psf/codeofconduct/

>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>> *Pronouns: he/him **(why is my pronoun here?)*
>>> 
>>>
>>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6AK5ZLHHZL5M2OTJUYQGEX3DSXBTH3N6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Multiple reference leaks in master

[Charalampos Stratakis]

- Original Message -
> From: "Pablo Galindo Salgado" 
> To: "Python Dev" 
> Sent: Tuesday, December 3, 2019 10:14:01 PM
> Subject: [Python-Dev] Multiple reference leaks in master
> 
> Hi,
> 
> Me (and Victor) have not been able to attend the buildbots for a while
> unfortunately and today
> I was checking them out after some fixes to the SSL tests and sadly the
> refleaks buildbots have many
> independent issues. At least these tests are failing due to reference leaks:
> 
> test__xxsubinterpreters, test_atexit, test_capi, test_httpservers,
> test_threading
> 
> I am trying to slowly fix them but finding them and debugging often takes
> several hours per leak. I was tracking them
> in https://bugs.python.org/issue38962 until I discover that there are
> multiple sources of leaks so I may open more issues.
> 
> Please, when you merge a PR, take a look at the buildbots (especially the
> refleak buildbots that do not report still to the PR
> in case of failure) so the issues don't keep piling up, as multiple refleaks
> together are much difficult to deal with.
> 
> Thanks, everyone!
> 
> Regards from cloudy London,
> Pablo Galindo Salgado
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6DWAFYZGH3X34VJHSOUVCEAHLZ4KZNPQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

Also noting here that the reference leaks builds are triggered once a day so 
many times it's not possible to find the commit that triggered a leak through 
the web ui of buildbot. In this case it's good to also check the previous 
builds of the same worker to figure out what changed.

-- 
Regards,

Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZP25JNHP5R5VIAQFYXTWANXSETW4V6MC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Steven D'Aprano]

I'm going to second Chris' comment about efficiency. The purposes of 
this PEP (as I read it) are:

(1) Security (less chance of code intentionally or accidentally 
exceeding low-level machine limits that allow a security exploit);

(2) Improved memory use;

(3) And such improved memory use will lead to faster code.

1 and 2 seem to be obviously true, but like Chris, I think its a bit 
much to expect us to take 3 on faith until after the PEP is accepted:

> Reference Implementation
> 
>
> None, as yet. This will be implemented in CPython, once the PEP has 
> been accepted.

I think the change you are asking for is akin to asking us to accept the 
GILectomy on the promise that "trust me, it will speed up CPython, no 
reference implementation is needed". It's a big thing to ask.

Most of us a Python programmers, not experts on the minutia of the 
interaction between C code and CPU cache locality and prediction etc. 
Speaking for myself, all I know is that it is *really hard* to predict 
what will and won't be faster:

improving memory locality will speed things up

but doing more work on every access will slow things down

so I'd like to see something more than just an assertion that this will 
speed things up.

Another question: how will this change effect CPython on less common 
CPUs like Atom etc?


As for the other objections I've seen so far, I think they are specious. 
(Sorry guys, I just think you are being knee-jerk naysayers. Convince me 
I am wrong.)

Obviously none of them is going to apply to hand-written code. Despite 
Rhodri's skepticism I don't think there is any really question of 
hand-written code hitting the limits of a million constants or a million 
local variables *per function*.

I just grepped the 3.8 stdlib for "class" and came up with fewer than 
22000 instances of the word:

[steve@ando cpython]$ grep -r "class" Lib/ | wc -l
21522

That includes docstrings, comments, the word "subclass" etc, but let's 
pretend that they're all actual classes. And let's round it up to 25000, 
and assume that there's another 25000 classes built into the 
interpreter, AND then *quadruple* that number to allow for third party 
libraries, that comes to just 250,000 classes. So we could load the 
entire stdlib and all our third-party libraries at once, and still be 
able to write 750,000 classes in your own code before hitting the limit.

Paddy: if you are generating a script from a log file, and it hits the 
million line boundary, it's easy to split it across multiple files. Your 
objection 

why limit a "quick hack"

has a simple answer: limiting that quick hack allows Python code to be 
quicker, more memory efficient and safer. If the cost of this is that 
you have to generate "quick hack" machine-generated Python scripts in 
multiple million-lines each files instead of one ginormous file, then 
that's a tradeoff well worth making.

Random832, I think the intention of this PEP is to specify limits for 
*CPython* specifically, not other implementations. Mark, can you 
clarify?

I don't understand why Steve Downer raises the issue of a million blank 
lines or comments. Machine generated code surely doesn't need blank 
lines. Blank lines could be stripped out; comments could be moved to 
another file. I see no real difficulty here.



-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3X3WGIJYBWGLBY6DUUJMBXWUKVSCQQ4C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Inada Naoki]

I think it is too early to determine when to remove it.
Even only talking about it causes blaming war.

BTW, I think 2to3 can help to move from 2&3 code to 3-only code.

* "future" fixer can be remove legacy futures.  But it seems to remove
all futures,
  including "annotations".  It should be kept.
* "unicode" fixer can be used to remove u-prefix.  But I'm not sure yet.

Are there any other things which are used for writing 2&3 code and
will be removed
someday in the future?
And is there any tool that can convert code using the "six" library to
normal Python 3 code?

Regards,

On Wed, Dec 4, 2019 at 2:29 AM Serhiy Storchaka  wrote:
>
> The 'u" string literal prefix was removed in 3.0 and reintroduced in 3.3
> to help writing the code compatible with Python 2 and 3 [1]. After the
> dead of Python 2.7 we will remove some deprecated features kept for
> compatibility with 2.7. When we are going to deprecate and remove the
> "u" prefix?
>
> [1] https://www.python.org/dev/peps/pep-0414/
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/EMFDKQ57JVWUZ6TPZM5VTFW7EUKVYAOY/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FDA2UHFY4PBZBWCUKU5HKM73KDKB7FT6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Kyle Stanley]

  > The number of live coroutines in a running interpreter.

Could you further elaborate on what is meant by "live coroutines"? My
guesses (roughly from most likely to least likely) would be:

1) All known coroutine objects in a state of either CORO_RUNNING or
CORO_SUSPENDED, but *not* CORO_CREATED or CORO_CLOSED.
2) Coroutine objects that are currently running, in a state of CORO_RUNNING
3) Coroutines objects being awaited, in a state of CORO_SUSPENDED
4) All known coroutine objects in a state of either CORO_CREATED,
CORO_RUNNING, or CORO_SUSPENDED
5) All known coroutine objects in any state
6) Total declared coroutines

Just so we're all on the same page, I'm referring to a "coroutine object"
as the object returned from the call `coro()`; whereas a "coroutine" is the
coroutine function/method `async def coro` (or the deprecated
generator-based coroutines).

It probably wouldn't be as much of a concern to only allow 1M running
coroutines at one time, but it might be an issue to only allow for there to
be 1M known coroutine objects in any state within a given interpreter.
Particularly for servers that run nearly indefinitely and handle a
significant number of concurrent requests.

On Tue, Dec 3, 2019 at 11:24 AM Mark Shannon  wrote:

> Hi Everyone,
>
> I am proposing a new PEP, still in draft form, to impose a limit of one
> million on various aspects of Python programs, such as the lines of code
> per module.
>
> Any thoughts or feedback?
>
> The PEP:
> https://github.com/markshannon/peps/blob/one-million/pep-100.rst
>
> Cheers,
> Mark.
>
>
> Full text
> *
>
> PEP: 100
> Title: The one million limit
> Author: Mark Shannon 
> Status: Active
> Type: Enhancement
> Content-Type: text/x-rst
> Created: 03-Dec-2019
> Post-History:
>
>
>
> Abstract
> 
> This PR proposes a limit of one million (1 000 000) for various aspects
> of Python code and its implementation.
>
> The Python language does not specify limits for many of its features.
> Not having any limit to these values seems to enhance programmer freedom,
> at least superficially, but in practice the CPython VM and other Python
> virtual
> machines have implicit limits or are forced to assume that the limits are
> astronomical, which is expensive.
>
> This PR lists a number of features which are to have a limit of one
> million.
> If a language feature is not listed but appears unlimited and must be
> finite,
> for physical reasons if no other, then a limit of one million should be
> assumed.
>
>
> Motivation
> ==
>
> There are many values that need to be represented in a virtual machine.
> If no limit is specified for these values, then the representation must
> either be inefficient or vulnerable to overflow.
> The CPython virtual machine represents values like line numbers,
> stack offsets and instruction offsets by 32 bit values. This is
> inefficient, and potentially unsafe.
>
> It is inefficient as actual values rarely need more than a dozen or so
> bits to represent them.
>
> It is unsafe as malicious or poorly generated code could cause values to
> exceed 2\ :sup:`32`.
>
> For example, line numbers are represented by 32 bit values internally.
> This is inefficient, given that modules almost never exceed a few
> thousand lines.
> Despite being inefficent, is is still vulnerable to overflow as
> it is easy for an attacker to created a module with billions of newline
> characters.
>
> Memory access is usually a limiting factor in the performance of modern
> CPUs.
> Better packing of data structures enhances locality and reduces memory
> bandwith,
> at a modest increase in ALU usage (for shifting and masking).
> Being able to safely store important values in 20 bits would allow
> memory savings
> in several data structures including, but not limited to:
>
> * Frame objects
> * Object headers
> * Code objects
>
> There is also the potential for a more efficient instruction format,
> speeding up interpreter dispatch.
>
> Rationale
> =
>
> Imposing a limit on values such as lines of code in a module, and the
> number of local variables,
> has significant advantages for ease of implementation and efficiency of
> virtual machines.
> If the limit is sufficiently large, there is no adverse effect on users
> of the language.
>
> By selecting a fixed but large limit for these values,
> it is possible to have both safety and efficiency whilst causing no
> inconvience to human programmers
> and only very rare problems for code generators.
>
> One million
> ---
>
> The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
> (65535) for many program
> elements similar to those covered here.
> This limit enables limited values to fit in 16 bits, which is a very
> efficient machine representation.
> However, this limit is quite easily exceeded in practice by code
> generators and
> the author is aware of existing Python code that already exceeds 2\
> :sup:`16` lines of code.
>
> A limit of one million fits 

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Gregory P. Smith]

On Tue, Dec 3, 2019 at 8:21 AM Mark Shannon  wrote:

> Hi Everyone,
>
> I am proposing a new PEP, still in draft form, to impose a limit of one
> million on various aspects of Python programs, such as the lines of code
> per module.
>
> Any thoughts or feedback?
>
> The PEP:
> https://github.com/markshannon/peps/blob/one-million/pep-100.rst
>
> Cheers,
> Mark.
>
>
> Full text
> *
>
> PEP: 100
> Title: The one million limit
> Author: Mark Shannon 
> Status: Active
> Type: Enhancement
> Content-Type: text/x-rst
> Created: 03-Dec-2019
> Post-History:
>
>
>
> Abstract
> 
> This PR proposes a limit of one million (1 000 000) for various aspects
> of Python code and its implementation.
>
> The Python language does not specify limits for many of its features.
> Not having any limit to these values seems to enhance programmer freedom,
> at least superficially, but in practice the CPython VM and other Python
> virtual
> machines have implicit limits or are forced to assume that the limits are
> astronomical, which is expensive.
>
> This PR lists a number of features which are to have a limit of one
> million.
> If a language feature is not listed but appears unlimited and must be
> finite,
> for physical reasons if no other, then a limit of one million should be
> assumed.
>
>
> Motivation
> ==
>
> There are many values that need to be represented in a virtual machine.
> If no limit is specified for these values, then the representation must
> either be inefficient or vulnerable to overflow.
> The CPython virtual machine represents values like line numbers,
> stack offsets and instruction offsets by 32 bit values. This is
> inefficient, and potentially unsafe.
>
> It is inefficient as actual values rarely need more than a dozen or so
> bits to represent them.
>
> It is unsafe as malicious or poorly generated code could cause values to
> exceed 2\ :sup:`32`.
>
> For example, line numbers are represented by 32 bit values internally.
> This is inefficient, given that modules almost never exceed a few
> thousand lines.
> Despite being inefficent, is is still vulnerable to overflow as
> it is easy for an attacker to created a module with billions of newline
> characters.
>
> Memory access is usually a limiting factor in the performance of modern
> CPUs.
> Better packing of data structures enhances locality and reduces memory
> bandwith,
> at a modest increase in ALU usage (for shifting and masking).
> Being able to safely store important values in 20 bits would allow
> memory savings
> in several data structures including, but not limited to:
>
> * Frame objects
> * Object headers
> * Code objects
>
> There is also the potential for a more efficient instruction format,
> speeding up interpreter dispatch.
>
> Rationale
> =
>
> Imposing a limit on values such as lines of code in a module, and the
> number of local variables,
> has significant advantages for ease of implementation and efficiency of
> virtual machines.
> If the limit is sufficiently large, there is no adverse effect on users
> of the language.
>
> By selecting a fixed but large limit for these values,
> it is possible to have both safety and efficiency whilst causing no
> inconvience to human programmers
> and only very rare problems for code generators.
>
> One million
> ---
>
> The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
> (65535) for many program
> elements similar to those covered here.
> This limit enables limited values to fit in 16 bits, which is a very
> efficient machine representation.
> However, this limit is quite easily exceeded in practice by code
> generators and
> the author is aware of existing Python code that already exceeds 2\
> :sup:`16` lines of code.
>
> A limit of one million fits into 20 bits which, although not as
> convenient for machine representation,
> is still reasonably compact. Three signed valuses in the range -1000_000
> to +1000_000 can fit into a 64 bit word.
> A limit of one million is small enough for efficiency advantages (only
> 20 bits),
> but large enough not to impact users (no one has ever written a module
> of one million lines).
>
> The value "one million" is very easy to remember.
>
> Isn't this "640K ought to be enough for anybody" again?
> ---
>
> The infamous 640K memory limit was a limit on machine usable resources.
> The proposed one million limit is a limit on human generated code.
>
> While it is possible that generated code could exceed the limit,
> it is easy for a code generator to modify its output to conform.
> The author has hit the 64K limit in the JVM on at least two occasions
> when generating Java code.
> The workarounds were relatively straightforward and
> probably wouldn't have been necessary with a limit of one million
> bytecodes or lines of code.
>
>
> Specification
> =
>
> This PR proposes that the following language features and runtime values
> be

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Ned Batchelder]

On 12/3/19 8:13 PM, Inada Naoki wrote:

I think it is too early to determine when to remove it.
Even only talking about it causes blaming war.


Has anyone yet given a reason to remove it? It will change working code 
into broken code. Why do that?


--Ned.



BTW, I think 2to3 can help to move from 2&3 code to 3-only code.

* "future" fixer can be remove legacy futures.  But it seems to remove
all futures,
   including "annotations".  It should be kept.
* "unicode" fixer can be used to remove u-prefix.  But I'm not sure yet.

Are there any other things which are used for writing 2&3 code and
will be removed
someday in the future?
And is there any tool that can convert code using the "six" library to
normal Python 3 code?

Regards,

On Wed, Dec 4, 2019 at 2:29 AM Serhiy Storchaka  wrote:

The 'u" string literal prefix was removed in 3.0 and reintroduced in 3.3
to help writing the code compatible with Python 2 and 3 [1]. After the
dead of Python 2.7 we will remove some deprecated features kept for
compatibility with 2.7. When we are going to deprecate and remove the
"u" prefix?

[1] https://www.python.org/dev/peps/pep-0414/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EMFDKQ57JVWUZ6TPZM5VTFW7EUKVYAOY/
Code of Conduct: http://python.org/psf/codeofconduct/




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/27OFJ45B4OFDR3ON2QI72XDF7ISVADU3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Chris Angelico]

On Wed, Dec 4, 2019 at 1:31 PM Gregory P. Smith  wrote:
> Overall I like the idea of limits... But... in my experience, limits like 
> this tend to impact generated source code or generated bytecode, and thus any 
> program that transitively uses those.
>

Overall, I *dislike* the idea of limits, but accept them when there's
a true and demonstrable benefit :)

I've worked with a lot of systems - languages (or
interpreters/compilers), file formats, etc, etc, etc - that have
arbitrary limits in them. The usual problem is that, a few years down
the track, what used to be "wow that's crazy huge" becomes "ugh now
I'm hitting this silly limit". For instance, PostgreSQL is limited to
1600 columns per table. Is that an insanely high limit that you'll
never hit, or an actual real limiting factor?

Integer sizes are a classic example of this. Is it acceptable to limit
your integers to 2^16? 2^32? 2^64? Python made the choice to NOT limit
its integers, and I haven't heard of any non-toy examples where an
attacker causes you to evaluate 2**2**100 and eats up all your RAM.
OTOH, being able to do arbitrary precision arithmetic and not worry
about an arbitrary limit to your precision is a very good thing.

IMO the limit is, in itself, a bad thing. If it's guaranteeing that
some exploit can't bring down your system, sure. If it permits a
significant and measurable performance benefit, sure. But the
advantage isn't the limit itself - it's what the limit enables. Which
I'd like to see more evidence of. :)

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GF4WOBEPJXMEMGATUMYR5A3D6H5E4FTM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] HighwayHash > SipHash?

[Larry Hastings]

My random Googling turned up a new hash function tonight: 
"HighwayHash".  It's a keyed hash function like the SipHash we now use 
for hashing strings / bytes / etc for our lovely dicts.


Key points:

 * Apache 2 license
 * Can use SIMD
 * "5x faster than SipHash"

I think they mean 5x faster than /their/ SipHash implementation, which 
they claim is an optimized implementation (but IIUC doesn't use SIMD).



Source is here:

   https://github.com/google/highwayhash

AFAICT they have multiple implementations to leverage different 
processor features, but one is vanilla portable C** so it should run 
everywhere Python 3 does.  In fact it already has a Python 3 package 
("pip3 install highwayhash-cffi") in case you want to play around with it.



Since the choice of SipHash is a private implementation detail, and 
since we all like fast things, is it worth considering switching to 
HighwayHash?  Don't ask me, I'm only a release manager (for a version 
nobody cares about anymore).  These things are above my pay grade.



Cheers,


//arry/

** They claim it's "C90", which I gather is C89 for Europeans hipsters 
who like obscure, withdrawn standards.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/26J6EEKJGXAX3USUCJSLX7KLGNFUVYUF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Steven D'Aprano]

On Wed, Dec 04, 2019 at 01:47:53PM +1100, Chris Angelico wrote:

> Integer sizes are a classic example of this. Is it acceptable to limit
> your integers to 2^16? 2^32? 2^64? Python made the choice to NOT limit
> its integers, and I haven't heard of any non-toy examples where an
> attacker causes you to evaluate 2**2**100 and eats up all your RAM.

Does self-inflicted attacks count? I've managed to bring down a 
production machine, causing data loss, *twice* by thoughtlessly running 
something like 10**100**100 at the interactive interpreter. (Neither 
case was a server, just a desktop machine, but the data loss was still 
very real.)


> OTOH, being able to do arbitrary precision arithmetic and not worry
> about an arbitrary limit to your precision is a very good thing.

I'll remind you of Guido's long-ago experience with ABC, which used 
arbitrary precision rationals (fractions) as their numeric type. That 
sounds all well and good, until you try doing a bunch of calculations 
and your numbers start growing to unlimited size. Do you really want a 
hundred billion digits of precision for a calculation based on 
measurements made to one decimal place?


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IAX2EFWOC4HSAUHZKD2Z4RUMUQZQZ7MH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Nathaniel Smith]

On Tue, Dec 3, 2019 at 8:20 AM Mark Shannon  wrote:
> The Python language does not specify limits for many of its features.
> Not having any limit to these values seems to enhance programmer freedom,
> at least superficially, but in practice the CPython VM and other Python
> virtual
> machines have implicit limits or are forced to assume that the limits are
> astronomical, which is expensive.

The basic idea makes sense to me. Well-defined limits that can be
supported properly are better than vague limits that are supported by
wishful thinking.

> This PR lists a number of features which are to have a limit of one
> million.
> If a language feature is not listed but appears unlimited and must be
> finite,
> for physical reasons if no other, then a limit of one million should be
> assumed.

This language is probably too broad... for example, there's certainly
a limit on how many objects can be alive at the same time due to the
physical limits of memory, but that limit is way higher than a
million.

> This PR proposes that the following language features and runtime values
> be limited to one million.
>
> * The number of source code lines in a module
> * The number of bytecode instructions in a code object.
> * The sum of local variables and stack usage for a code object.
> * The number of distinct names in a code object
> * The number of constants in a code object.

These are all attributes of source files, so sure, a million is
plenty, and the interpreter spends a ton of time manipulating tables
of these things.

> * The number of classes in a running interpreter.

This one isn't as obvious to me... classes are basically just objects
of type 'type', and there is definitely code out there that creates
classes dynamically. A million still seems like a lot, and I'm not
saying I'd *recommend* a design that involves creating millions of
different type objects, but it might exist already.

> * The number of live coroutines in a running interpreter.

I don't get this one. I'm not thinking of any motivation (the
interpreter doesn't track live coroutines differently from any other
object), and the limit seems dangerously low. A million coroutines
only requires a few gigabytes of RAM, and there are definitely people
who run single process systems with >1e6 concurrent tasks (random
example: https://goroutines.com/10m)

I don't know if there's anyone doing this in *Python right now, due to
Python's performance limitations, but it's nowhere near as silly as a
function with a million local variables.

> Total number of classes in a running interpreter
> 
>
> This limit has to the potential to reduce the size of object headers
> considerably.
>
> Currently objects have a two word header, for objects without references
> (int, float, str, etc.) or a four word header for objects with references.
> By reducing the maximum number of classes, the space for the class reference
> can be reduced from 64 bits to fewer than 32 bits allowing a much more
> compact header.
>
> For example, a super-compact header format might look like this:
>
> .. code-block::
>
>  struct header {
>  uint32_t gc_flags:6; /* Needs finalisation, might be part of a
> cycle, etc. */
>  uint32_t class_id:26; /* Can be efficiently mapped to address
> by ensuring suitable alignment of classes */
>  uint32_t refcount; /* Limited memory or saturating */
>  }
>
> This format would reduce the size of a Python object without slots, on a
> 64 bit machine, from 40 to 16 bytes.

In this example, I can't figure out how you'd map your 26 bit class_id
to a class object. On a 32-bit system it would be fine, you just need
64 byte alignment, but you're talking about 64-bit systems, so... I
know you aren't suggesting classes should have 2**(64 - 26) =
~3x10**11 byte alignment :-)

-n

--
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TL62SYQ6DGCCLRTIGMCUFAT5UEWMB7KN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Inada Naoki]

On Wed, Dec 4, 2019 at 11:49 AM Ned Batchelder  wrote:
>
> On 12/3/19 8:13 PM, Inada Naoki wrote:
> > I think it is too early to determine when to remove it.
> > Even only talking about it causes blaming war.
>
> Has anyone yet given a reason to remove it?

Note that "never" is included in ”when".
I didn't promoting removing it.  I just said let's stop discussion about it.


> It will change working code
> into broken code.  Why do that?

Of course, everyone in this thread understands it.
No one proposes remove it now.

On the other hand, we remove some parts from Python language
and the stdlib regularly to keep Python clean.
All removal will break some code.  That's why we have a deprecation period.

Currently, u-prefix is very widely used.  It shouldn't be removed anytime soon.
And I agree that we shouldn't raise DeprecationWarning for now.

But how about 5, 10, and 20 years later?  No one knows it.
Let's stop discussing it.  It can not be productive.

Instead, we can do:

* Don't recommend u-prefix except in Python 2&3 code.
* Provide a tool to remove the u-prefix.

Regards,
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EVKCCO5KMOGEEFMSSY2PZRVGT2LDOB5K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Chris Angelico]

On Wed, Dec 4, 2019 at 3:16 PM Steven D'Aprano  wrote:
>
> On Wed, Dec 04, 2019 at 01:47:53PM +1100, Chris Angelico wrote:
>
> > Integer sizes are a classic example of this. Is it acceptable to limit
> > your integers to 2^16? 2^32? 2^64? Python made the choice to NOT limit
> > its integers, and I haven't heard of any non-toy examples where an
> > attacker causes you to evaluate 2**2**100 and eats up all your RAM.
>
> Does self-inflicted attacks count? I've managed to bring down a
> production machine, causing data loss, *twice* by thoughtlessly running
> something like 10**100**100 at the interactive interpreter. (Neither
> case was a server, just a desktop machine, but the data loss was still
> very real.)

Hmm, and you couldn't Ctrl-C it? I tried and was able to.

There ARE a few situations where I'd rather get a simple and clean
MemoryError than have it drive my system into the swapper, but there
are at least as many situations where you'd rather be able to use
virtual memory instead of being forced to manually break a job up. But
even there, you can't enshrine a limit in the language definition,
since the actual threshold depends on the running system. (And can be
far better enforced externally, at least on a Unix-like OS.)

> > OTOH, being able to do arbitrary precision arithmetic and not worry
> > about an arbitrary limit to your precision is a very good thing.
>
> I'll remind you of Guido's long-ago experience with ABC, which used
> arbitrary precision rationals (fractions) as their numeric type. That
> sounds all well and good, until you try doing a bunch of calculations
> and your numbers start growing to unlimited size. Do you really want a
> hundred billion digits of precision for a calculation based on
> measurements made to one decimal place?
>

Sometimes, yes! But if I don't, it's more likely that I want to choose
a limit within the program, rather than run into a hard limit defined
by the language. I've done a lot of work with fractions.Fraction and
made good use of its immense precision.

The Python float type gives a significant tradeoff in terms of
performance vs precision. But decimal.Decimal lets you choose exactly
how much precision to retain, rather than baking it into the language
as "no more than 1,000,000 digits of precision, ever". The solution to
"do you really want a hundred billion digits of precision" is "use the
decimal context to choose", not "hard-code a limit". The value of the
hard-coded limit in a float is that floats are way WAY faster than
Decimals.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UIUG2BIZSES7REMRICGSYA2UJ2PFT52O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Kyle Stanley]

> BTW, I think 2to3 can help to move from 2&3 code to 3-only code.

> Instead, we can do:

> * Don't recommend u-prefix except in Python 2&3 code.
> * Provide a tool to remove the u-prefix.

+1, this seems like the smoothest way of handling it and has very minimal
impact on users. In 5+ years from now as u-strings become increasingly
rare, we can consider a long term deprecation process. In the meantime, any
tools we can provide to automate the process of converting them will make
the transition that much easier.

On Wed, Dec 4, 2019 at 12:41 AM Inada Naoki  wrote:

> On Wed, Dec 4, 2019 at 11:49 AM Ned Batchelder 
> wrote:
> >
> > On 12/3/19 8:13 PM, Inada Naoki wrote:
> > > I think it is too early to determine when to remove it.
> > > Even only talking about it causes blaming war.
> >
> > Has anyone yet given a reason to remove it?
>
> Note that "never" is included in ”when".
> I didn't promoting removing it.  I just said let's stop discussion about
> it.
>
>
> > It will change working code
> > into broken code.  Why do that?
>
> Of course, everyone in this thread understands it.
> No one proposes remove it now.
>
> On the other hand, we remove some parts from Python language
> and the stdlib regularly to keep Python clean.
> All removal will break some code.  That's why we have a deprecation period.
>
> Currently, u-prefix is very widely used.  It shouldn't be removed anytime
> soon.
> And I agree that we shouldn't raise DeprecationWarning for now.
>
> But how about 5, 10, and 20 years later?  No one knows it.
> Let's stop discussing it.  It can not be productive.
>
> Instead, we can do:
>
> * Don't recommend u-prefix except in Python 2&3 code.
> * Provide a tool to remove the u-prefix.
>
> Regards,
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EVKCCO5KMOGEEFMSSY2PZRVGT2LDOB5K/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AKUXJR6BO4WTO7SLN3ES6PCWNO5H62ZU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Chris Angelico]

On Wed, Dec 4, 2019 at 5:22 PM Kyle Stanley  wrote:
>
> > BTW, I think 2to3 can help to move from 2&3 code to 3-only code.
>
> > Instead, we can do:
>
> > * Don't recommend u-prefix except in Python 2&3 code.
> > * Provide a tool to remove the u-prefix.
>
> +1, this seems like the smoothest way of handling it and has very minimal 
> impact on users. In 5+ years from now as u-strings become increasingly rare, 
> we can consider a long term deprecation process. In the meantime, any tools 
> we can provide to automate the process of converting them will make the 
> transition that much easier.
>

The first one is already the case. PEP 414 reintroduced the u"..."
literal form, specifically as a porting tool. Given that it has
absolutely zero value in pure Py3 code, it can be assumed to be not
recommended outside of 2/3 compatibility.

I'm not sure how much value there'd be in a tool *just* to remove the
u prefix, but perhaps a suite of 2to3-like fixers for "hey, now that
you're dropping Python 2 support, here's some things you can take
advantage of". For instance, "class Foo(object):" can become "class
Foo:", and many uses of super() can become the zero-argument form. +1
on getting such a tool started.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3CEPZAFTMNSRWZ43GV5W3BKU6GJZEIVT/
Code of Conduct: http://python.org/psf/codeofconduct/