[Numpy-discussion] Dropping PyPy in NumPy 2.3
Hi All, Just thought I would raise the topic of PyPy status in NumPy 2.3. PyPy for Python 3.11 currently has a bug which has hung around for a while, and it is also likely that there will be no PyPy for Python 3.12. It looks like PyPy is reaching end of life, and this seems a good time to decide if we should continue releasing wheels for it. Thoughts? Chuck ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
- Mail original - > De: "Stefan Krah" > À: "numpy-discussion" > Envoyé: Mercredi 7 Mai 2025 13:47:19 > Objet: [Numpy-discussion] Re: Better compatibility of the Python > scientific/data stack with fast Python interpreters > On Wed, Apr 30, 2025 at 02:08:36PM +0200, PIERRE AUGIER wrote: >> It seems to me that a strategy based on HPy would give practical benefices >> for >> users in a much shorter time (typically few years) than just waiting for >> CPython C API evolution. > > It would be nice to use alternative interpreters, but I still see conflicting > messages about the performance of HPy: > > https://github.com/psycopg/psycopg/issues/154 > > "If the goal is to run psycopg3 fast in PyPy, there are at least three paths: > >use a cffi backend: this is likely to be the fastest one >(in case psycopg uses Cython and doesn't call directly any CPython C API): > use >the Cython HPy backend, once it's ready >(in case psycopg uses the CPython C API directly): rewrite it to use HPy >instead." One can create from a C file using the HPy API a standard Python extension using in the background the CPython C API. So for CPython, there should be no performance penalty. Numpy-HPy should be on CPython as fast as current Numpy. From the same file, one can also create a universal extension that can be used on different Python implementations (and for different Python versions). On CPython, there is a relatively small performance penalty (few percents). On PyPy and GraalPy, the calls are basically as fast as on CPython with the standard Python extension. Then, there can be other performance improvements with other methods, some already possible (in particular on GraalPy) and some that would need further HPy developments (avoiding boxing/unboxing). > CFFI however is slower, e.g. for _decimal, than the native CPython C-API. > (At least it was in version 3.9, _decimal has been slowed down significantly > since I left.) Could you explain a bit more? Or provide a link? CFFI with PyPy? or CFFI with CPython? Is there a proper reproducible benchmark? > _decimal of course operates on scalars and has many API calls, so maybe > for NumPy this is not relevant except for small arrays. Yes, for most real world code using Numpy, the performance of the interface is not crucial for the overall program performance. It is just important to avoid very inefficient things, like the emulation layers needed with the CPython C API (like PyPy cpyext). > Or perhaps HPy has evolved in the meantime (the above GitHub thread is from > 2022). > > > Stefan Krah > > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: pierre.aug...@univ-grenoble-alpes.fr ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
On Wed, May 07, 2025 at 03:52:33PM +0200, PIERRE AUGIER via NumPy-Discussion
wrote:
> > "If the goal is to run psycopg3 fast in PyPy, there are at least three
> > paths:
> >
> >use a cffi backend: this is likely to be the fastest one
> >(in case psycopg uses Cython and doesn't call directly any CPython C
> > API): use
> >the Cython HPy backend, once it's ready
> >(in case psycopg uses the CPython C API directly): rewrite it to use HPy
> >instead."
[cut]
> Could you explain a bit more? Or provide a link? CFFI with PyPy? or CFFI with
> CPython?
Yes: _if_ the above 2022 quote from the psycopg3 issue is still valid,
and my benchmarks show that the CPython C-API is faster than the PyPy
CFFI, then (according to the quote), HPy is slower than PyPy CFFI and
therefore much slower than CPython C-API.
All of that under the assumption that there are many API calls.
Stefan Krah
> Is there a proper reproducible benchmark?
Here are the benchmarks for _decimal:
bench.py
import time
import platform
if platform.python_implementation() == "CPython":
import _decimal as C
else: # PyPy
import __decimal as C
def pi(D):
lasts, t, s, n, na, d, da = D(0), D(3), D(3), D(1), D(0), D(0), D(24)
while s != lasts:
lasts = s
n, na = n+na, na+8
d, da = d+da, da+32
t = (t * n) / d
s += t
return s
print("\n#
==")
print("# Calculating pi, 1 iterations")
print("#
==\n")
prec = 9
print("\nPrecision: %d decimal digits\n" % prec)
start = time.time()
C.getcontext().prec = prec
for i in range(1):
x = pi(C.Decimal)
print("result: %s" % str(x))
print("time: %fs\n" % (time.time()-start))
### NOTE: only execute git clean -xdf in a throwaway repository! ###
git clone https://github.com/python/cpython
cd cpython
git checkout v3.9.0
./configure && make -j32
cd ..
./cpython/python bench.py
[cut]
time: 0.169412s
cd cpython
git clean -xdf
git checkout v3.13.0
./configure && make -j32
cd ..
./cpython/python bench.py
[cut]
time: 0.238405s # Massive slowdown compared to 3.9.
wget https://downloads.python.org/pypy/pypy3.11-v7.3.19-linux64.tar.bz2
tar xvf pypy3.11-v7.3.19-linux64.tar.bz2
cd pypy3.11-v7.3.19-linux64/lib/pypy3.11/
../../bin/pypy _decimal_build.py # Builds _decimal_cffi using libmpdec.
cd ../../..
./pypy3.11-v7.3.19-linux64/bin/pypy bench.py
[cut]
time: 2.433634s
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters
On Wed, Apr 30, 2025 at 02:08:36PM +0200, PIERRE AUGIER wrote: > It seems to me that a strategy based on HPy would give practical benefices > for users in a much shorter time (typically few years) than just waiting for > CPython C API evolution. It would be nice to use alternative interpreters, but I still see conflicting messages about the performance of HPy: https://github.com/psycopg/psycopg/issues/154 "If the goal is to run psycopg3 fast in PyPy, there are at least three paths: use a cffi backend: this is likely to be the fastest one (in case psycopg uses Cython and doesn't call directly any CPython C API): use the Cython HPy backend, once it's ready (in case psycopg uses the CPython C API directly): rewrite it to use HPy instead." CFFI however is slower, e.g. for _decimal, than the native CPython C-API. (At least it was in version 3.9, _decimal has been slowed down significantly since I left.) _decimal of course operates on scalars and has many API calls, so maybe for NumPy this is not relevant except for small arrays. Or perhaps HPy has evolved in the meantime (the above GitHub thread is from 2022). Stefan Krah ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Dropping PyPy in NumPy 2.3
On Wed, May 7, 2025 at 7:44 PM Charles R Harris via NumPy-Discussion < numpy-discussion@python.org> wrote: > Hi All, > > Just thought I would raise the topic of PyPy status in NumPy 2.3. PyPy for > Python 3.11 currently has a bug which has hung around for a while, > Is it relevant? There are of course multiple open issues for PyPy, but none that seem critical or are failing in CI. > and it is also likely that there will be no PyPy for Python 3.12. It looks > like PyPy is reaching end of life, and this seems a good time to decide if > we should continue releasing wheels for it. > We're already near the date for 2.3.x branch creation, so if PyPy isn't an extra burden I'd keep it for one more release. We could drop it in `main` after the branch is created - that removes most of the support burden, and may anyway be warranted as either 2.4.x or 2.5.x is going to be the release series that drops Python 3.11 support. Cheers, Ralf > > Thoughts? > > Chuck > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: ralf.gomm...@googlemail.com > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
