[Python-Dev] Re: Resurrecting PEP-472
First draft is available as a draft PR on https://github.com/python/peps/pull/1579 This draft contains just the motivation, not the implementation decisions. I'll add that in the upcoming nights, but it's now open for comments. On Mon, 31 Aug 2020 at 22:50, Stefano Borini wrote: > > On Sat, 29 Aug 2020 at 19:33, Antoine Pitrou wrote: > > > Agreed with Victor. Please create a new PEP. > > on it. > > -- > Kind regards, > > Stefano Borini -- Kind regards, Stefano Borini ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4VTPGPLTMR5ZUKQXDTCJEUYZAO7Y35SN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Doc tests failing for many PRs on GitHub
Hi, The documentation job of our pre-commit CI fails because of a change in the newly released setuptools 50.0: it comes from the new _distutil_hack module of setuptools. I proposed https://github.com/pypa/setuptools/pull/2361 to fix Python 3.10 support in the _distutils_hack module. Sadly, in the meanwhile, it's no longer possible to merge any PR in Python since the documentation job is mandatory. Either we wait until setuptools is fixed, or we make the documentation job non-voting. I propose to wait until setuptools is fixed and use this time to review a bunch of pending pull requests :-) Victor Le lun. 31 août 2020 à 19:30, Ammar Askar a écrit : > > Filed https://github.com/pypa/setuptools/issues/2362 against > setuptools, not sure what we should do locally to fix this in the > meantime. > > On Mon, Aug 31, 2020 at 1:11 PM Ammar Askar wrote: > > > > Karthik, your analysis is correct. You just have to import importlib > > first to get the error: > > > > >>> import importlib > > >>> import importlib.util > > >>> importlib.util.abc.Loader > > Traceback (most recent call last): > > File "", line 1, in > > AttributeError: module 'importlib.util' has no attribute 'abc' > > > > On Mon, Aug 31, 2020 at 12:09 PM Karthikeyan wrote: > > > > > > The last successful build is > > > https://travis-ci.com/github/python/cpython/jobs/379161961 > > > Recent failure build is > > > https://travis-ci.com/github/python/cpython/jobs/379329436 > > > > > > I can see setuptools being upgraded from 49.6.0 to 50.0.0 as a > > > difference. I can see the below command to print sphinx-build version > > > error out in the makefile but unfortunately the error is being redirected > > > to /dev/null in the makefile and I needed to run it explicitly to see the > > > output. There is a recent change that moves Loader to _abc and imports > > > Loader from _abc inside abc is done with below commit. The same code is > > > present in 49.6.0 too at > > > https://github.com/pypa/setuptools/blob/04e3df22df840c6bb244e9b27bc56750c44b7c85/_distutils_hack/__init__.py#L79 > > > . So I am not sure why this causes error. > > > > > > Pinning the dependency of setuptools to 49.6.0 in Doc/makefile runs the > > > tests fine without error. Can you please try pinning it and see if it > > > helps? > > > > > > I am not sure if the below commit is the cause that is picked up in > > > setuptools but someone can correct me if I am wrong about the importlib > > > machinery error. > > > > > > commit 9e09849d20987c131b28bcdd252e53440d4cd1b3 > > > Author: Victor Stinner > > > Date: Wed Jun 17 23:15:59 2020 +0200 > > > > > > bpo-41006: importlib.util no longer imports typing (GH-20938) > > > > > > Create importlib._abc submodule to avoid importing typing when > > > importlib.util is imported. Move Loader ABC into importlib._abc. > > > > > > ./python > > > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > > > [GCC 7.5.0] on linux > > > Type "help", "copyright", "credits" or "license" for more information. > > > >>> import importlib.util > > > >>> importlib.util.abc.Loader > > > Traceback (most recent call last): > > > File "", line 1, in > > > AttributeError: module 'importlib.util' has no attribute 'abc' > > > >>> > > > ➜ cpython git:(master) ✗ git checkout > > > 9e09849d20987c131b28bcdd252e53440d4cd1b3~1 Lib/importlib/util.py > > > ➜ cpython git:(master) ✗ ./python > > > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > > > [GCC 7.5.0] on linux > > > Type "help", "copyright", "credits" or "license" for more information. > > > >>> import importlib.util > > > >>> importlib.util.abc.Loader > > > > > > > > > xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going > > > -j4" doctest suspicious html > > > make: Entering directory '/root/cpython/Doc' > > > make[1]: Entering directory '/root/cpython/Doc' > > > mkdir -p build > > > echo PATH=./venv/bin:$PATH blurb help > > > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > > blurb help > > > echo PATH=./venv/bin:$PATH sphinx-build --version > > > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > > sphinx-build --version > > > Building NEWS from Misc/NEWS.d with blurb > > > PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -q -W > > > --keep-going -j4 -W . build/doctest > > > Traceback (most recent call last): > > > File "/root/cpython/Doc/./venv/bin/sphinx-build", line 5, in > > > from sphinx.cmd.build import main > > > File > > > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/cmd/build.py", > > > line 23, in > > > from sphinx.application import Sphinx > > > File > > > "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/application.py", > > > line 28, in > > > from sphinx.config import Config > > > File > > > "/roo
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
Hi, On Tue, 01 Sep 2020 02:52:07 - "Raihan Rasheed Apurbo" wrote: > In CPython we have reference counting. My question is can we optimize current > RC using > strategies like Deferred RC and Coalescing? If no then where would I face > problems if I try > to implement these sorts of strategies? You've already asked your question on python-ideas and already got answers there. In any case, the only definitive response to "would I face problems" can be obtained by trying it out. So, if you've got some time on your hands, I would encourage experimenting and share your results here. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CGFCS46KUAFUUY3UQMSDDYPPX5NDEWF7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Doc tests failing for many PRs on GitHub
I've opened https://bugs.python.org/issue41685 and pushed a temporary change to the Docs build venv to pin the setuptools version to a previous version. I've also restarted the failing CI test for Mark's PRs and a couple of others. On Sep 1, 2020, at 04:35, Victor Stinner wrote: > The documentation job of our pre-commit CI fails because of a change > in the newly released setuptools 50.0: it comes from the new > _distutil_hack module of setuptools. > > I proposed https://github.com/pypa/setuptools/pull/2361 to fix Python > 3.10 support in the _distutils_hack module. > > Sadly, in the meanwhile, it's no longer possible to merge any PR in > Python since the documentation job is mandatory. Either we wait until > setuptools is fixed, or we make the documentation job non-voting. > > I propose to wait until setuptools is fixed and use this time to > review a bunch of pending pull requests :-) > > Victor > > Le lun. 31 août 2020 à 19:30, Ammar Askar a écrit : >> >> Filed https://github.com/pypa/setuptools/issues/2362 against >> setuptools, not sure what we should do locally to fix this in the >> meantime. >> >> On Mon, Aug 31, 2020 at 1:11 PM Ammar Askar wrote: >>> >>> Karthik, your analysis is correct. You just have to import importlib >>> first to get the error: >>> >> import importlib >> import importlib.util >> importlib.util.abc.Loader >>> Traceback (most recent call last): >>> File "", line 1, in >>> AttributeError: module 'importlib.util' has no attribute 'abc' >>> >>> On Mon, Aug 31, 2020 at 12:09 PM Karthikeyan wrote: The last successful build is https://travis-ci.com/github/python/cpython/jobs/379161961 Recent failure build is https://travis-ci.com/github/python/cpython/jobs/379329436 I can see setuptools being upgraded from 49.6.0 to 50.0.0 as a difference. I can see the below command to print sphinx-build version error out in the makefile but unfortunately the error is being redirected to /dev/null in the makefile and I needed to run it explicitly to see the output. There is a recent change that moves Loader to _abc and imports Loader from _abc inside abc is done with below commit. The same code is present in 49.6.0 too at https://github.com/pypa/setuptools/blob/04e3df22df840c6bb244e9b27bc56750c44b7c85/_distutils_hack/__init__.py#L79 . So I am not sure why this causes error. Pinning the dependency of setuptools to 49.6.0 in Doc/makefile runs the tests fine without error. Can you please try pinning it and see if it helps? I am not sure if the below commit is the cause that is picked up in setuptools but someone can correct me if I am wrong about the importlib machinery error. commit 9e09849d20987c131b28bcdd252e53440d4cd1b3 Author: Victor Stinner Date: Wed Jun 17 23:15:59 2020 +0200 bpo-41006: importlib.util no longer imports typing (GH-20938) Create importlib._abc submodule to avoid importing typing when importlib.util is imported. Move Loader ABC into importlib._abc. ./python Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) [GCC 7.5.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import importlib.util >>> importlib.util.abc.Loader Traceback (most recent call last): File "", line 1, in AttributeError: module 'importlib.util' has no attribute 'abc' >>> ➜ cpython git:(master) ✗ git checkout 9e09849d20987c131b28bcdd252e53440d4cd1b3~1 Lib/importlib/util.py ➜ cpython git:(master) ✗ ./python Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) [GCC 7.5.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import importlib.util >>> importlib.util.abc.Loader xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going -j4" doctest suspicious html make: Entering directory '/root/cpython/Doc' make[1]: Entering directory '/root/cpython/Doc' mkdir -p build echo PATH=./venv/bin:$PATH blurb help PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games blurb help echo PATH=./venv/bin:$PATH sphinx-build --version PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games sphinx-build --version Building NEWS from Misc/NEWS.d with blurb PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -q -W --keep-going -j4 -W . build/doctest Traceback (most recent call last): File "/root/cpython/Doc/./venv/bin/sphinx-build", line 5, in from sphinx.cmd.build import main File "/root/cpython/Doc/venv/lib/python3.10/site-packages/sphinx/cmd/build.py",
[Python-Dev] Re: Doc tests failing for many PRs on GitHub
On Sep 1, 2020, at 05:47, Ned Deily wrote: > I've opened https://bugs.python.org/issue41685 and pushed a temporary change > to the Docs build venv to pin the setuptools version to a previous version. > I've also restarted the failing CI test for Mark's PRs and a couple of others. But, of course, just restarting the failed CI test isn't going to work right away since the PRs don't include the new setuptools pin. So either cause the PRs to be updated against the current HEAD of master by pushing new commits to them or wait for a fixed version of setuptools and then cause the CI to rerun. I'll let the affected PR authors take care of that :) > On Sep 1, 2020, at 04:35, Victor Stinner wrote: >> The documentation job of our pre-commit CI fails because of a change >> in the newly released setuptools 50.0: it comes from the new >> _distutil_hack module of setuptools. >> >> I proposed https://github.com/pypa/setuptools/pull/2361 to fix Python >> 3.10 support in the _distutils_hack module. >> >> Sadly, in the meanwhile, it's no longer possible to merge any PR in >> Python since the documentation job is mandatory. Either we wait until >> setuptools is fixed, or we make the documentation job non-voting. >> >> I propose to wait until setuptools is fixed and use this time to >> review a bunch of pending pull requests :-) >> >> Victor >> >> Le lun. 31 août 2020 à 19:30, Ammar Askar a écrit : >>> >>> Filed https://github.com/pypa/setuptools/issues/2362 against >>> setuptools, not sure what we should do locally to fix this in the >>> meantime. >>> >>> On Mon, Aug 31, 2020 at 1:11 PM Ammar Askar wrote: Karthik, your analysis is correct. You just have to import importlib first to get the error: >>> import importlib >>> import importlib.util >>> importlib.util.abc.Loader Traceback (most recent call last): File "", line 1, in AttributeError: module 'importlib.util' has no attribute 'abc' On Mon, Aug 31, 2020 at 12:09 PM Karthikeyan wrote: > > The last successful build is > https://travis-ci.com/github/python/cpython/jobs/379161961 > Recent failure build is > https://travis-ci.com/github/python/cpython/jobs/379329436 > > I can see setuptools being upgraded from 49.6.0 to 50.0.0 as a > difference. I can see the below command to print sphinx-build version > error out in the makefile but unfortunately the error is being redirected > to /dev/null in the makefile and I needed to run it explicitly to see the > output. There is a recent change that moves Loader to _abc and imports > Loader from _abc inside abc is done with below commit. The same code is > present in 49.6.0 too at > https://github.com/pypa/setuptools/blob/04e3df22df840c6bb244e9b27bc56750c44b7c85/_distutils_hack/__init__.py#L79 > . So I am not sure why this causes error. > > Pinning the dependency of setuptools to 49.6.0 in Doc/makefile runs the > tests fine without error. Can you please try pinning it and see if it > helps? > > I am not sure if the below commit is the cause that is picked up in > setuptools but someone can correct me if I am wrong about the importlib > machinery error. > > commit 9e09849d20987c131b28bcdd252e53440d4cd1b3 > Author: Victor Stinner > Date: Wed Jun 17 23:15:59 2020 +0200 > > bpo-41006: importlib.util no longer imports typing (GH-20938) > > Create importlib._abc submodule to avoid importing typing when > importlib.util is imported. Move Loader ABC into importlib._abc. > > ./python > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > [GCC 7.5.0] on linux > Type "help", "copyright", "credits" or "license" for more information. import importlib.util importlib.util.abc.Loader > Traceback (most recent call last): > File "", line 1, in > AttributeError: module 'importlib.util' has no attribute 'abc' > ➜ cpython git:(master) ✗ git checkout > 9e09849d20987c131b28bcdd252e53440d4cd1b3~1 Lib/importlib/util.py > ➜ cpython git:(master) ✗ ./python > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > [GCC 7.5.0] on linux > Type "help", "copyright", "credits" or "license" for more information. import importlib.util importlib.util.abc.Loader > > > xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going > -j4" doctest suspicious html > make: Entering directory '/root/cpython/Doc' > make[1]: Entering directory '/root/cpython/Doc' > mkdir -p build > echo PATH=./venv/bin:$PATH blurb help > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > blurb help > echo PATH=./venv/bin:$PATH sphinx-build --version > PATH=./venv/bin:/root/.poetry/bin:/usr/local/sb
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
Before experimenting can't I ask someone whether relevant experiments were made or not? if I am not supposed to do that then I totally agree with you, sir. I tried asking in python-ideas and one of the replies was "I gather there have been some experiments along these lines as part of efforts to remove the GIL. You might like to research them to find out how much success they've had." What I can interpret from this reply is there are some experiments regarding this topic but I don't know where they are. Then I asked him where could I find them but didn't get any answer to that. Then how have I got my answers? I am so sorry if I am asking any inappropriate question here. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2IC27QRCFB6CSVMMYR3NTVU34TI5J6XN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
On Tue, Sep 1, 2020 at 9:25 AM Raihan Rasheed Apurbo
wrote:
> Before experimenting can't I ask someone whether relevant experiments were
> made or not? if I am not supposed to do that then I totally agree with
> you, sir.
>
> I tried asking in python-ideas and one of the replies was "I gather there
> have been some experiments along these lines as part
> of efforts to remove the GIL. You might like to research them to find out
> how much success they've had."
>
> What I can interpret from this reply is there are some experiments
> regarding this topic but I don't know where they are. Then I asked him
> where could I find them but didn't get any answer to that.
>
> Then how have I got my answers?
>
> I am so sorry if I am asking any inappropriate question here.
>
Raihan,
Nobody knows the answers to your questions ("can we optimize...", "where
would I face problems...", "is there something I am missing?", "tell me the
factors I should consider").
Your questions are somehow presented as if we *should* know the answers,
and seem to insinuate that we should have investigated the techniques you
mention. The answer to this is that we are doubtful that it will be easy to
implement them in a fully backwards compatible way (and you'd be surprised
how important backwards compatibility is).
The only thing I personally know about this topic is that Larry Hastings
attempted a more sophisticated approach to multi-threaded reference
counting without the GIL, and came up short despite putting a lot of effort
in it. You should probably do some Googling on Gilectomy and find Larry's
various annual updates. That is presumably what that python-ideas answer
was referring to.
Good luck.
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
Search for “Larry Hastings gilectomy”. For example, there’s this: https://speakerdeck.com/pycon2017/larry-hastings-the-gilectomy-hows-it-going -- Eric V. Smith (301) 502-0945 cell > On Sep 1, 2020, at 12:21 PM, Raihan Rasheed Apurbo wrote: > > Before experimenting can't I ask someone whether relevant experiments were > made or not? if I am not supposed to do that then I totally agree with you, > sir. > > I tried asking in python-ideas and one of the replies was "I gather there > have been some experiments along these lines as part > of efforts to remove the GIL. You might like to research them to find out how > much success they've had." > > What I can interpret from this reply is there are some experiments regarding > this topic but I don't know where they are. Then I asked him where could I > find them but didn't get any answer to that. > > Then how have I got my answers? > > I am so sorry if I am asking any inappropriate question here. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/2IC27QRCFB6CSVMMYR3NTVU34TI5J6XN/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ULP7CU2OQPBCY7FGVTOFC6LKXNSCFTIW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
In today's CPython with the GIL, it's hard to beat the existing simple
reference counting approach. It's already so cheap, and CPython's
implementation has had a lot of optimization.
I implemented "buffered reference counting" in the Gilectomy because
it's far friendlier to multicore. It added an enormous amount of
overhead. But with enough cores it would eventually become a
performance win.
I'm not optimistic that coalescing would be a win for CPython. On the C
side, reference count change code is written by hand, so hopefully it
already does an intelligent job without redundant reference changes. On
the bytecode side, reference count changes are implicit, so there's
nothing you can optimize. Changing CPython bytecode to make explicit
reference count changes would add so much overhead--the generated
bytecode would be much larger, you'd be dispatching so many reference
count change opcodes--that it's hard to imagine it would be cheaper than
just living with the redundant reference count changes.
My memory is hazy, but IIRC "deferred" reference counting is a clever
scheme in which stack references don't change the reference count, only
heap references do. An object has to have a reference count of zero
/and/ have no active references in the stack in order to be collected.
This seems like a lovely idea, maximizing the best attributes of both
reference counting and tracing garbage collection. However, this would
require walking the stack(s), which we can't do in cross-platform ANSI
C. It would also require a complete overhaul of the C code for CPython
itself (and eventually all extensions), hand-reviewing every reference
count change to decide "is this a heap or a stack reference"?
I don't think there is "low-hanging fruit" left to optimize in CPython's
reference counting scheme. So all that's left are major rewrites like
"deferred" reference counting. Personally I think the future of CPython
is to change completely over to tracing garbage collection. It's so
much friendlier to multicore, which is clearly the future of
programming. I'd rather see efforts in this area directed towards that
goal.
//arry/
On 9/1/20 9:34 AM, Guido van Rossum wrote:
On Tue, Sep 1, 2020 at 9:25 AM Raihan Rasheed Apurbo
mailto:apurb...@gmail.com>> wrote:
Before experimenting can't I ask someone whether relevant
experiments were made or not? if I am not supposed to do that
then I totally agree with you, sir.
I tried asking in python-ideas and one of the replies was "I
gather there have been some experiments along these lines as part
of efforts to remove the GIL. You might like to research them to
find out how much success they've had."
What I can interpret from this reply is there are some experiments
regarding this topic but I don't know where they are. Then I
asked him where could I find them but didn't get any answer to that.
Then how have I got my answers?
I am so sorry if I am asking any inappropriate question here.
Raihan,
Nobody knows the answers to your questions ("can we optimize...",
"where would I face problems...", "is there something I am missing?",
"tell me the factors I should consider").
Your questions are somehow presented as if we *should* know the
answers, and seem to insinuate that we should have investigated the
techniques you mention. The answer to this is that we are doubtful
that it will be easy to implement them in a fully backwards compatible
way (and you'd be surprised how important backwards compatibility is).
The only thing I personally know about this topic is that Larry
Hastings attempted a more sophisticated approach to multi-threaded
reference counting without the GIL, and came up short despite putting
a lot of effort in it. You should probably do some Googling on
Gilectomy and find Larry's various annual updates. That is presumably
what that python-ideas answer was referring to.
Good luck.
--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
On 2020-09-01, Larry Hastings wrote: > Personally I think the future of CPython is to change completely > over to tracing garbage collection. It's so much friendlier to > multicore, which is clearly the future of programming. I'd rather > see efforts in this area directed towards that goal. I think either CPython does that or some other implementation is going to displace it. CPython doesn't have a good way of utilizing multi-core CPUs. The various multi-process approaches don't solve the problem of efficiently passing data between threads of execution. An elegant approach would be to use message passing like is done by Erlang. However, given that Python is not a functional language and that most core data structures are mutable, it seems a poor fit. The most obvious approach is to adopt a multi-threaded model like is done by modern Java. I.e. no GIL and non-thread safe core data structures. That sounds a bit scary but based on Java experience it seems programmers can manage it. If it wasn't for CPython's reference counted GC, that kind of threading model seems relatively easy to implement. Getting libgc working would be a useful first step: https://discuss.python.org/t/switching-from-refcounting-to-libgc/1641 I guess it's hard to get people excited about that work because you have to go backwards in performance before you can possibily go forward. The non-reference counted CPython is going to be much slower and recovering that performance will be a long slog. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VBBPHP3FNJQHJLLESO2BTUFJ63H5KSS5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Doc tests failing for many PRs on GitHub
Thanks for reverting the setuptools version Ned, and to Victor for opening a PR to make the fix for the latest version. I'm always amazed by the efforts made and quick responses to keep things running smoothly. :-) On Tue, Sep 1, 2020 at 5:56 AM Ned Deily wrote: > On Sep 1, 2020, at 05:47, Ned Deily wrote: > > I've opened https://bugs.python.org/issue41685 and pushed a temporary > change to the Docs build venv to pin the setuptools version to a previous > version. I've also restarted the failing CI test for Mark's PRs and a > couple of others. > > But, of course, just restarting the failed CI test isn't going to work > right away since the PRs don't include the new setuptools pin. So either > cause the PRs to be updated against the current HEAD of master by pushing > new commits to them or wait for a fixed version of setuptools and then > cause the CI to rerun. > > I'll let the affected PR authors take care of that :) > > > > On Sep 1, 2020, at 04:35, Victor Stinner wrote: > >> The documentation job of our pre-commit CI fails because of a change > >> in the newly released setuptools 50.0: it comes from the new > >> _distutil_hack module of setuptools. > >> > >> I proposed https://github.com/pypa/setuptools/pull/2361 to fix Python > >> 3.10 support in the _distutils_hack module. > >> > >> Sadly, in the meanwhile, it's no longer possible to merge any PR in > >> Python since the documentation job is mandatory. Either we wait until > >> setuptools is fixed, or we make the documentation job non-voting. > >> > >> I propose to wait until setuptools is fixed and use this time to > >> review a bunch of pending pull requests :-) > >> > >> Victor > >> > >> Le lun. 31 août 2020 à 19:30, Ammar Askar a > écrit : > >>> > >>> Filed https://github.com/pypa/setuptools/issues/2362 against > >>> setuptools, not sure what we should do locally to fix this in the > >>> meantime. > >>> > >>> On Mon, Aug 31, 2020 at 1:11 PM Ammar Askar > wrote: > > Karthik, your analysis is correct. You just have to import importlib > first to get the error: > > >>> import importlib > >>> import importlib.util > >>> importlib.util.abc.Loader > Traceback (most recent call last): > File "", line 1, in > AttributeError: module 'importlib.util' has no attribute 'abc' > > On Mon, Aug 31, 2020 at 12:09 PM Karthikeyan > wrote: > > > > The last successful build is > https://travis-ci.com/github/python/cpython/jobs/379161961 > > Recent failure build is > https://travis-ci.com/github/python/cpython/jobs/379329436 > > > > I can see setuptools being upgraded from 49.6.0 to 50.0.0 as a > difference. I can see the below command to print sphinx-build version error > out in the makefile but unfortunately the error is being redirected to > /dev/null in the makefile and I needed to run it explicitly to see the > output. There is a recent change that moves Loader to _abc and imports > Loader from _abc inside abc is done with below commit. The same code is > present in 49.6.0 too at > https://github.com/pypa/setuptools/blob/04e3df22df840c6bb244e9b27bc56750c44b7c85/_distutils_hack/__init__.py#L79 > . So I am not sure why this causes error. > > > > Pinning the dependency of setuptools to 49.6.0 in Doc/makefile runs > the tests fine without error. Can you please try pinning it and see if it > helps? > > > > I am not sure if the below commit is the cause that is picked up in > setuptools but someone can correct me if I am wrong about the importlib > machinery error. > > > > commit 9e09849d20987c131b28bcdd252e53440d4cd1b3 > > Author: Victor Stinner > > Date: Wed Jun 17 23:15:59 2020 +0200 > > > > bpo-41006: importlib.util no longer imports typing (GH-20938) > > > > Create importlib._abc submodule to avoid importing typing when > > importlib.util is imported. Move Loader ABC into importlib._abc. > > > > ./python > > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > > [GCC 7.5.0] on linux > > Type "help", "copyright", "credits" or "license" for more > information. > import importlib.util > importlib.util.abc.Loader > > Traceback (most recent call last): > > File "", line 1, in > > AttributeError: module 'importlib.util' has no attribute 'abc' > > > ➜ cpython git:(master) ✗ git checkout > 9e09849d20987c131b28bcdd252e53440d4cd1b3~1 Lib/importlib/util.py > > ➜ cpython git:(master) ✗ ./python > > Python 3.10.0a0 (heads/master:75c80b0bda, Aug 30 2020, 13:53:05) > > [GCC 7.5.0] on linux > > Type "help", "copyright", "credits" or "license" for more > information. > import importlib.util > importlib.util.abc.Loader > > > > > > xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W > --keep-going -j4" doctest suspicious html > > make: Entering directory '/root/cpython/Doc' > > make[1]: Entering dire
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
On 2/09/20 8:32 am, Neil Schemenauer wrote: The most obvious approach is to adopt a multi-threaded model like is done by modern Java. I.e. no GIL and non-thread safe core data structures. That sounds a bit scary but based on Java experience it seems programmers can manage it. I think that depends on how non-thread-safe it is. If it's "weird things can happen" it might be all right. But if it's "can crash the interpreter" it might not be all right. -- Greg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GKD455W5AIWO42N54GPFGPCHCAA4UMZ7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
On 2020-09-02, Greg Ewing wrote: > On 2/09/20 8:32 am, Neil Schemenauer wrote: > > The most obvious approach is to adopt a multi-threaded model like is > > done by modern Java. I.e. no GIL and non-thread safe core data > > structures. That sounds a bit scary but based on Java experience it > > seems programmers can manage it. > > I think that depends on how non-thread-safe it is. If it's > "weird things can happen" it might be all right. But if > it's "can crash the interpreter" it might not be all right. Weird things would include unexpected exceptions. This seems a relevant discussion: https://softwareengineering.stackexchange.com/questions/262428/race-conditions-in-jvm-languages-versus-c-c#262440 The Java spec seems to contain the details but I admit I haven't studied them: https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html Getting exceptions if your locking is incorrect seems an okay tradeoff to me. My knowledge of Java is pretty limited but I believe they originally tried to make the core data structures thread-safe (e.g. maps, vectors). That turned out to be too difficult or too expensive. Instead, the core collection types are not thread-safe and they introduced new "concurrent" collections. That way, you only pay the cost of synchronization if you need it. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JSYPXRC2N4LXVYIP4IB53LQKKFDYLD4M/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
On 2/09/20 6:11 pm, Neil Schemenauer wrote: Weird things would include unexpected exceptions. I don't mind unexpected exceptions, but I would mind core dumps. -- Greg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/U5YW6KXPYHIO6DG6HYE6KOUWTAV7MLF6/ Code of Conduct: http://python.org/psf/codeofconduct/
