[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Antoine Pitrou]

On Sat, 11 Apr 2020 02:11:41 +0200
Victor Stinner  wrote:
> Le ven. 10 avr. 2020 à 22:00, Antoine Pitrou  a écrit :
> > > Debug runtime and remove debug checks in release mode
> > > .
> > >
> > > If the C extensions are no longer tied to CPython internals, it becomes
> > > possible to switch to a Python runtime built in debug mode to enable
> > > runtime debug checks to ease debugging C extensions.  
> >
> > That's the one convincing feature in this PEP, as far as I'm concerned.  
> 
> In fact, I already implemented this feature in Python 3.8:
> https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

I had missed that. Great! :-)

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MW3K7AHKDVWMKJCOFFUK5DR3GIDPG6WK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Antoine Pitrou]

On Sat, 11 Apr 2020 01:52:13 +0200
Victor Stinner  wrote:
> 
> By the way, CPython currently uses statically allocated types for
> builtin types like str or list. This may have to change to run
> efficiently multiple subinterepters in parallel: each subinterpeter
> should have its own heap-allocated type with its own reference
> counter.
> 
> Using heap allocated types means that PyUnicode_Check() implementation
> has to change. It's just another good reason to better hide
> PyUnicode_Check() implementation right now ;-)

I'm not sure I understand.  If PyUnicode_Check() uses tp_flags, it
doesn't have to change, precisely.

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QKBWU2IQSGYKSU3SQZ5N25KJ47ZDXOIE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Rhodri James]

On 10/04/2020 18:20, Victor Stinner wrote:

Note: Cython and cffi should be preferred to write new C extensions.
This PEP is about existing C extensions which cannot be rewritten with
Cython.


If this is true, the documentation on python.org needs a serious 
rewrite.  I am in the throes of writing a C extension, and using Cython 
or cffi never even crossed my mind.


--
Rhodri James *-* Kynesim Ltd
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NEYKZMQ6ZWB5VGS22SZTVQPVOJETDKUQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Ivan Pozdeev via Python-Dev]

On 10.04.2020 20:20, Victor Stinner wrote:

Hi,

Here is a first draft a PEP which summarize the research work I'm
doing on CPython C API since 2017 and the changes that me and others
already made since Python 3.7 towards an "opaque" C API. The PEP is
also a collaboration with developers of PyPy, HPy, Rust-CPython and
many others! Thanks to everyone who helped me to write it down!

Maybe this big document should be reorganized as multiple smaller
better defined goals: as multiple PEPs. The PEP is quite long and
talks about things which are not directly related. It's a complex
topic and I chose to put everything as a single document to have a
good starting point to open the discussion. I already proposed some of
these ideas in 2017: see the Prior Art section ;-)

The PEP can be read on GitHub where it's better formatted:
https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst

If someone wants to work on the PEP itself, the document on GitHub is
the current reference.

Victor



PEP xxx: Modify the C API to hide implementation details


Abstract


* Hide implementation details from the C API to be able to `optimize
   CPython`_ and make PyPy more efficient.
* The expectation is that `most C extensions don't rely directly on
   CPython internals`_ and so will remain compatible.
* Continue to support old unmodified C extensions by continuing to
   provide the fully compatible "regular" CPython runtime.
* Provide a `new optimized CPython runtime`_ using the same CPython code
   base: faster but can only import C extensions which don't use
   implementation details. Since both CPython runtimes share the same
   code base, features implemented in CPython will be available in both
   runtimes.
* `Stable ABI`_: Only build a C extension once and use it on multiple
   Python runtimes and different versions of the same runtime.
* Better advertise alternative Python runtimes and better communicate on
   the differences between the Python language and the Python
   implementation (especially CPython).

Note: Cython and cffi should be preferred to write new C extensions.
This PEP is about existing C extensions which cannot be rewritten with
Cython.


Rationale
=

To remain competitive in term of performance with other programming
languages like Go or Rust, Python has to become more efficient.

Make Python (at least) two times faster
---

The C API leaks too many implementation details which prevent optimizing
CPython. See `Optimize CPython`_.

PyPy's support for Python's C API (pycext) is slow because it has to
emulate CPython internals like memory layout and reference counting. The
emulation causes memory overhead, memory copies, conversions, etc. See
`Inside cpyext: Why emulating CPython C API is so Hard
`_
(Sept 2018) by Antonio Cuni.

While this PEP may make CPython a little bit slower in the short term,
the long-term goal is to make "Python" at least two times faster. This
goal is not hypothetical: PyPy is already 4.2x faster than CPython and is
fully compatible. C extensions are the bottleneck of PyPy. This PEP
proposes a migration plan to move towards opaque C API which would make
PyPy faster.

Separated the Python language and the CPython runtime (promote
alternative runtimes)


The Python language should be better separated from its runtime. It's
common to say "Python" when referring to "CPython". Even in this PEP :-)

Because the CPython runtime remains the reference implementation, many
people believe that the Python language itself has design flaws which
prevent it from being efficient. PyPy proved that this is a false
assumption: on average, PyPy runs Python code 4.2 times faster than
CPython.

One solution for separating the language from the implementation is to
promote the usage of alternative runtimes: not only provide the regular
CPython, but also PyPy, optimized CPython which is only compatible with
C extensions using the limited C API, CPython compiled in debug mode to
ease debugging issues in C extensions, RustPython, etc.

To make alternative runtimes viable, they should be competitive in term
of features and performance. Currently, C extension modules remain the
bottleneck for PyPy.

Most C extensions don't rely directly on CPython internals
--

While the C API is still tidely coupled to CPython internals, in
practical, most C extensions don't rely directly on CPython internals.

The expectation is that these C extensions will remain compatible with
an "opaque" C API and only a minority of C extensions will have to be
modified.

Moreover, more and more C extensions are implemented in Cython or cffi.

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Phil Thompson]

On 11/04/2020 13:08, Ivan Pozdeev via Python-Dev wrote:

On 10.04.2020 20:20, Victor Stinner wrote:



Stable ABI
--

The idea is to build a C extension only once: the built binary will be
usable on multiple Python runtimes and different versions of the same
runtime (stable ABI).

The idea is not new but is an extension of the `PEP 384: Defining a
Stable ABI `__ implemented 
in

CPython 3.4 with its "limited C API". The limited API is not used by
default and is not widely used: PyQt is one of the only few known 
users.


The idea here is that the default C API becomes the limited C API and 
so

all C extensions will benefit of advantages of a stable ABI.


In my practice with helping maintain a C extension module, it's not a
problem to build the module separately for every minor release.

That's because there are only a few officially supported releases, and
they aren't released frequently.

Conversely, if you are using a "limited ABI", you are "limited" (pun
intended) to what it has and can't take advantage of any new features
until the next major Python version -- i.e. for potentially several
years!

So I don't see any "advantages of a stable ABI" atm that matter in
practice while I do see _dis_advantages. So this area can perhaps be
excluded from the PEP or at least given low priority.
Unless, of course, you have some other, more real upcoming "advantages" 
in mind.


PyQt uses the stable ABI because it dramatically reduces the number of 
wheels that need to be created for a full release.


PyQt consists of 6 different PyPI packages. Wheels are provided for 4 
different platforms. Currently Python v3.5 to v3.8 are supported.


With the stable ABI that's 24 wheels for a full release. No additional 
wheels are needed when Python v3.9 is supported.


Without the stable ABI it would be 96 wheels. 24 additional wheels would 
be needed when Python v3.9 is supported.


Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GJLDX6SYWLOJ7JFAE6LCJZ6WEQJAYRGG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Python 2 death march

[Sumana Harihareswara]

Thanks. I'm working to get https://github.com/python/steering-council/issues/3 
resolved by April 17th to add an informational header to all the deep links 
within https://docs.python.org/2/* . I welcome help on the pull requests linked 
from that issue (such as https://github.com/python/cpython/pull/19229 ), and on 
the question Leonard Richardson asks there regarding 
https://github.com/python/devguide/blob/master/devcycle.rst .
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S5Y6422FVG27LJ5PQXKJXRV567WKZTPZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Providing fix for modulefinder.py regression on Windows

[Barry Scott]

modulefinder.py does not open source files in "rb" which
prevents compile() from applying the encoding rules.

This first showed up for me on Windows with Python 3.8.

I'd like to provide patches for 3.8 (if any more releases)
and 3.9.

What is the process I should follow?
I have signed the Contributor Agreement.

Here is my test case and the results on Windows with 3.7.

import modulefinder

with open('import_functools.py', 'w') as f:
f.write('import functools\n')

mf = modulefinder.ModuleFinder()
mf.run_script('import_functools.py')
print('Test passed')

py -3.8-32 modulefinder_test.py
Traceback (most recent call last):
  File "modulefinder_test.py", line 7, in 
mf.run_script('import_functools.py')
  File "C:\Python38.win32\lib\modulefinder.py", line 165, in run_script
self.load_module('__main__', fp, pathname, stuff)
  File "C:\Python38.win32\lib\modulefinder.py", line 360, in load_module
self.scan_code(co, m)
  File "C:\Python38.win32\lib\modulefinder.py", line 433, in scan_code
self._safe_import_hook(name, m, fromlist, level=0)
  File "C:\Python38.win32\lib\modulefinder.py", line 378, in _safe_import_hook
self.import_hook(name, caller, level=level)
  File "C:\Python38.win32\lib\modulefinder.py", line 177, in import_hook
q, tail = self.find_head_package(parent, name)
  File "C:\Python38.win32\lib\modulefinder.py", line 233, in find_head_package
q = self.import_module(head, qname, parent)
  File "C:\Python38.win32\lib\modulefinder.py", line 326, in import_module
m = self.load_module(fqname, fp, pathname, stuff)
  File "C:\Python38.win32\lib\modulefinder.py", line 343, in load_module
co = compile(fp.read()+'\n', pathname, 'exec')
  File "C:\Python38.win32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 308: 
character maps to 

Barry
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6IWG2SXTZAWEWYIBQ5LUR5G5S7MUODMV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Providing fix for modulefinder.py regression on Windows

[Guido van Rossum]

This seems like a noble pursuit. File a bug on bugs.python.org with the
results you posted here, and submit a PR via GitHub (
github.com/python/cpython/).

On Sat, Apr 11, 2020 at 9:09 AM Barry Scott  wrote:

> modulefinder.py does not open source files in "rb" which
> prevents compile() from applying the encoding rules.
>
> This first showed up for me on Windows with Python 3.8.
>
> I'd like to provide patches for 3.8 (if any more releases)
> and 3.9.
>
> What is the process I should follow?
> I have signed the Contributor Agreement.
>
> Here is my test case and the results on Windows with 3.7.
>
> import modulefinder
>
> with open('import_functools.py', 'w') as f:
> f.write('import functools\n')
>
> mf = modulefinder.ModuleFinder()
> mf.run_script('import_functools.py')
> print('Test passed')
>
> py -3.8-32 modulefinder_test.py
> Traceback (most recent call last):
>   File "modulefinder_test.py", line 7, in 
> mf.run_script('import_functools.py')
>   File "C:\Python38.win32\lib\modulefinder.py", line 165, in run_script
> self.load_module('__main__', fp, pathname, stuff)
>   File "C:\Python38.win32\lib\modulefinder.py", line 360, in load_module
> self.scan_code(co, m)
>   File "C:\Python38.win32\lib\modulefinder.py", line 433, in scan_code
> self._safe_import_hook(name, m, fromlist, level=0)
>   File "C:\Python38.win32\lib\modulefinder.py", line 378, in
> _safe_import_hook
> self.import_hook(name, caller, level=level)
>   File "C:\Python38.win32\lib\modulefinder.py", line 177, in import_hook
> q, tail = self.find_head_package(parent, name)
>   File "C:\Python38.win32\lib\modulefinder.py", line 233, in
> find_head_package
> q = self.import_module(head, qname, parent)
>   File "C:\Python38.win32\lib\modulefinder.py", line 326, in import_module
> m = self.load_module(fqname, fp, pathname, stuff)
>   File "C:\Python38.win32\lib\modulefinder.py", line 343, in load_module
> co = compile(fp.read()+'\n', pathname, 'exec')
>   File "C:\Python38.win32\lib\encodings\cp1252.py", line 23, in decode
> return codecs.charmap_decode(input,self.errors,decoding_table)[0]
> UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position
> 308: character maps to 
>
> Barry
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6IWG2SXTZAWEWYIBQ5LUR5G5S7MUODMV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EIBRS7XK7T7EFAZV4CQX6I3ASIA7RTLK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Fixing Copy on Writes with Immortal Instances

[Eddie Elizondo]

Copy on writes are a big problem in large Python application that rely on 
multiple processes sharing the same memory.

I just sent out a patch introduces Immortal Instances which allows the user to 
bypass reference counting on specific objects to avoid CoW on forked processes. 
This has helped in reducing the memory constraints in some of our applications.

The PR has a more in-depth explanation about the change. Looking forward to get 
your thoughts.

Github PR: https://github.com/python/cpython/pull/19474
Bug Report: https://bugs.python.org/issue40255


- Eddie E.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CKXZE5NZWCR3BCQGP33BWKJ7MND5TD2Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python2 as 𝑣 → 𝑒

[Mike Miller]

Unless I've read something wrong, it looks like the final Python 2 release 
(2.7.18) should approximate the math constant e:


>>> import math
>>> math.e
2.718281828459045

Aka:

Python as 𝑣 → 𝑒
Python ≈ 𝑒

Would be fun to note that in the announcement/docs somehow.

-Mike
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HXJLVHYYRBQKZ43HGYM3LSZDGXNOOMWC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python2 as 𝑣 → 𝑒

[Kacvinsky, Tom]

> -Original Message-
> From: Mike Miller 
> Sent: Saturday, April 11, 2020 6:17 PM
> To: python-dev@python.org
> Subject: [Python-Dev] Python2 as 𝑣 → 𝑒
> 
> 
> Unless I've read something wrong, it looks like the final Python 2 release
> (2.7.18) should approximate the math constant e:
> 
>  >>> import math
>  >>> math.e
>  2.718281828459045
> 
> Aka:
> 
>  Python as 𝑣 → 𝑒
>  Python ≈ 𝑒
> 
> Would be fun to note that in the announcement/docs somehow.

Of note, metafont already has the distinction of being versioned after the 
digits of e,
just like TeX is versioned after the digits of pi.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NOVIBPPVXFSY7RDGMM3GULQPVDAV3X7U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python2 as 𝑣 → 𝑒

[Benjamin Peterson]

The relevant parties are aware.

On Sat, Apr 11, 2020, at 17:17, Mike Miller wrote:
> 
> Unless I've read something wrong, it looks like the final Python 2 release 
> (2.7.18) should approximate the math constant e:
> 
>  >>> import math
>  >>> math.e
>  2.718281828459045
> 
> Aka:
> 
>  Python as 𝑣 → 𝑒
>  Python ≈ 𝑒
> 
> Would be fun to note that in the announcement/docs somehow.
> 
> -Mike
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/HXJLVHYYRBQKZ43HGYM3LSZDGXNOOMWC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GT52CV3DFJ5DGFCSTXPG7XRAZFVRJF57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python2 as 𝑣 → 𝑒

[Steven D'Aprano]

On Sat, Apr 11, 2020 at 03:17:07PM -0700, Mike Miller wrote:

> Python as 𝑣 → 𝑒
> Python ≈ 𝑒

For anyone else who had trouble seeing those glyphs, the ASCII 
equivalent is:

Python as v --> e
Python ~= e

with the v and e in italics and the ~= meaning "approximately equal".


-- 
Steven
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2BIVGJZAWMDKHKMXZ2OAKLHRQ7OR6KFC/
Code of Conduct: http://python.org/psf/codeofconduct/