[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
On Fri, Jan 28, 2022 at 6:28 PM Guido van Rossum wrote:
> I think we will get *one* chance in the next decade to get it right. Whether
> that's HPy or evolution of the C API I'm not sure.
Would you mind to elaborate? Which risk do you expect from switching
to HPy and from fixing the C API (introducing incompatible C API
changes)?
For me, promoting HPy and evolution of the C API are complementary,
can and must done in parallel for me. As I explained in PEP 674, while
HPy does help C extensions writers, it doesn't solve any problem for
CPython right now. CPython is still blocked by implementation details
leaked throught the C API that we must still maintain for a few more
years.
> Victor, am I right that the (some) stable ABI will remain important because
> projects don't have resources to build wheels for every Python release? If a
> project does R releases per year for P platforms that need to support V
> versions of Python, they would normally have to build R * P * V wheels. With
> a stable ABI, they could reduce that to R * P. That's the key point, right?
There are different use cases.
1) First, my main worry is that we put a high pressure on maintainers
of most important Python dependencies before the next of a new Python
version, because we want them to handle the flow of incompatible C API
changes before the final Python 3.x versions is released, to get them
available when Python 3.x final is released.
It annoys core developers who cannot change things in Python without
getting an increasing number of complains about a large number of
broken packages, sometimes with a request to revert.
It annoys C extensions maintainers who have to care about Python alpha
and beta releases which are not convenient to use (ex: not available
in Linux distributions). Moreover, it became common to ask multiple
changes and multiple releases before a Python final release, since
more incompatible changes are introduced in Python (before the beta1).
2) Second, as you said, the stable ABI reduces the number of binary
packages which have to be built. Small projects with a little team
(ex: a single person) don't have resources to set up a CI and maintain
it to build all these packages. It's doable, but it isn't free.
--
The irony of the situation is that we must break the C API (hiding
structures is technically an incompatible change)... to make the C API
stable. Breaking it now to make it stable later.
We already broke the C API many times in the past. The difference here
is that changes are done in the purpose of bending it towards the
limited C API and the stable ABI.
My expectation is that replacing frame->f_code with PyFrame_GetCode()
only has to be done exactly once: this API is not going this change.
Sadly, the changes are not limited to frame->f_code, more changes are
needed. For example, for PyFrameObject, accesses to every structure
member must have to go through a function call (getter or setter
function). Hopefully, only a small number of member are used by C
extensions.
The tricky part is to think about the high level API ("use cases")
rather than just adding functions doing "return struct->member" and
"struct->member = new_value". The PyThreadState_EnterTracing() and
PyThreadState_LeaveTracing() functions added to Python 3.11 are a good
example: the API is "generic" and the implementation changes 2
structure members, not a single one.
In practice, what I did since Python 3.8 is to introduce a small
number of C API changes per Python versions. We tried the "fix all the
things at once" approach (!!!) with Python 3, and it... didn't go
well. All C extensions had to suddenly write their own compatibility
layer for a large number of C API functions (ex: replace PyInt_xxx
with PyLong_xxx, without losing Python 2 support!). The changes that
I'm introducing in the C API usually impact less than 100 extensions
in total (usually, I would say between 10 and 25 per Python version,
but it's hard to measure exactly).
> Can HPy do that?
I wish more projects are incrementally rewritten with Cython, cffi,
pybind11 and HPy, and so slowly move away using directly the C API.
Yes, HPy support an "universal build" mode which allows to only build
a C extension once, and use it on multiple *CPython* versions *and*
(that's the big news!) multiple *PyPy* versions! I even heard that it
also brings GraalPython support for free ;-)
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/4I7N3SCBWIJCYXE3WGCQBMRTATVGIHY6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Blocking the main branch due to too many buildbots failing
On Sat, Jan 29, 2022 at 1:01 AM Pablo Galindo Salgado wrote: > Hi everyone, > > We have a huge amount of buildbots failing and seems to be related to > different issues > that keep piling up. To prevent this from going worse,* I am blocking the > main branch* > *until these issues are resolved.* > > Only release managers and the developer in residence will be able to merge > pull requests > until these issues are fixed. Please, ping me if you have a pull request > for fixing any of these > issues so we can merge. > > I apologize for the inconvenience. > > Thanks for your understanding, > > Regards from rainy London, > Pablo Galindo Salgado > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/HU2LH5MX44QZXZ36YS4SOG4ZUPT4E7QT/ > Code of Conduct: http://python.org/psf/codeofconduct/ > Noting here that I've recently added some CentOS Stream 9 buildbots to the fleet, however many are failing unfortunately. Since they are new additions and the failures haven't been ironed out, I've created a PR to move them to unstable as to not block anything, till things get sorted out: https://github.com/python/buildmaster-config/pull/308 -- Regards, Charalampos Stratakis Senior Software Engineer Python Maintenance Team, Red Hat ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HOY3K4NJGJCQEQ4FVWHHAZXCHD5VQRW2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Increase of Spammy PRs and PR reviews
As someone who is watching the python/cpython repository, I'm very used to see lots of traffic. But lately there have been a surge of spammy PRs which are about the same, generally very trivial subject but individually fixing each occurrence one by one: - Add coverage to X (tens of them, as separate PRs) - Make `test_xxx` executable with direct invocation (tens of them, as separate PRs) - Lint source with flake8, fix linting errors (stylistic changes) And lots of non-committer PR reviews that only approve. Am I the only one who feels like this is only done to 'grind' commits (get a higher number of commits into the repository, and boast about it)? In the past there were one or two people who would submit typo fixes, but most of them weren't making it continuously. The situation right now feels much worse than those.___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7RGI4LUJSEKU2JUYSV7UMZ2CXRGAANEF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Increase of Spammy PRs and PR reviews
On 1/29/22 3:14 AM, Lrupert via Python-Dev wrote: > As someone who is watching the python/cpython repository, I'm very used to see lots of traffic. But > lately there have been a surge of spammy PRs which are about the same, generally very trivial subject > but individually fixing each occurrence one by one: > - Add coverage to X (tens of them, as separate PRs) > - Make `test_xxx` executable with direct invocation (tens of them, as separate PRs) > - Lint source with flake8, fix linting errors (stylistic changes) I am aware of the coverage PRs, and I think they are quite useful -- a couple bugs were fixed in one of my modules thanks to that author's efforts to make sure all branches were tested. Likewise, the executable test PRs are useful -- I haven't followed them closely, but at the beginning a few of us asked the author to submit one PR for a handful of related fixes, which makes it easier for reviewers to assess the changes. So those two types are not spam. I am unaware of the stylistic changes -- could you list a few? > And lots of non-committer PR reviews that only approve. I have seen this. Quite irritating. > In the past there were one or two people who would submit typo fixes, but most of them weren't making it continuously. > The situation right now feels much worse than those. As I said earlier, the coverage and executable tests are legitimate, and I welcome them -- they are helping increase the quality of Python's code. -- ~Ethan~ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TCRMIQ6ZVVDOO3MIVHT5WWUWDEY3A3J3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Increase of Spammy PRs and PR reviews
On Sun, Jan 30, 2022 at 10:39 AM Ethan Furman wrote: > > > > And lots of non-committer PR reviews that only approve. > > I have seen this. Quite irritating. > We can prohibit approval from non core developers. Do we use this setting already? https://github.blog/2021-11-01-github-keeps-getting-better-for-open-source-maintainers/ -- Inada Naoki ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KHXSE2MSEC5JR2QB5F6HJUFYCC4SHGFF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Increase of Spammy PRs and PR reviews
As a non-committer, I want to make a plea for non-committer approval reviews, or at least that they have a place. When asked how outsiders can contribute I frequently see "review open PRs" as a suggested way of contributing to CPython. Not being able to approve PRs that are good would be a barrier to those contributions. Furthermore, I am collaborating with a couple of core devs, it would make collaboration more difficult if I couldn't review their work and say that I thought the changes looked good. I know "drive by approvals" are annoying but I think it is unfortunately part of open source projects. On Sat, Jan 29, 2022, 5:51 PM Inada Naoki wrote: > On Sun, Jan 30, 2022 at 10:39 AM Ethan Furman wrote: > > > > > > > And lots of non-committer PR reviews that only approve. > > > > I have seen this. Quite irritating. > > > > We can prohibit approval from non core developers. Do we use this > setting already? > > https://github.blog/2021-11-01-github-keeps-getting-better-for-open-source-maintainers/ > > > -- > Inada Naoki > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/KHXSE2MSEC5JR2QB5F6HJUFYCC4SHGFF/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BKIUK2KGP2YMN7GBEYYQVTEMECE2WTSD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Increase of Spammy PRs and PR reviews
On Sun, Jan 30, 2022 at 12:03 PM Ethan Smith wrote: > > As a non-committer, I want to make a plea for non-committer approval reviews, > or at least that they have a place. When asked how outsiders can contribute I > frequently see "review open PRs" as a suggested way of contributing to > CPython. Not being able to approve PRs that are good would be a barrier to > those contributions. > > Furthermore, I am collaborating with a couple of core devs, it would make > collaboration more difficult if I couldn't review their work and say that I > thought the changes looked good. > You can still write a review comment, including "LGTM". What you can not is labeling PR as "Approved." So I don't think it makes collaboration difficult. By preventing approval from others, we can easily find PRs approved from core-devs or triage members but not merged yet. > I know "drive by approvals" are annoying but I think it is unfortunately part > of open source projects. > Sorry, but I don't think so. -- Inada Naoki ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/X4PIFVMHMIECGAEP7A2SKMUXZD4BVWIM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] RFC on PEP 655: Required[] and NotRequired[] for TypedDict
This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as required or not-required. We have implementations for mypy, pyright, and pyanalyze. Consensus on the PEP has been reached in typing-sig. My understanding of the PEP process as described in PEP 1 [2] is that content review should be requested of core developers here in python-dev. So consider this my official request for comments! ^_^ I believe that after review & feedback from python-dev the next step would be to submit this PEP to the Steering Council. However it's not clear to me from [2] where I should actually do that when the time comes. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy [1]: https://www.python.org/dev/peps/pep-0655/ [2]: https://www.python.org/dev/peps/pep-0001/#pep-review-resolution ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/C7QCZ5J277O5WHDJF5PFTA453DY7DS7S/ Code of Conduct: http://python.org/psf/codeofconduct/
