[Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods
Hello, Let me present PEP 579 and PEP 580. PEP 579 is an informational meta-PEP, listing some of the issues with functions/methods implemented in C. The idea is to create several PEPs each fix some part of the issues mentioned in PEP 579. PEP 580 is a standards track PEP to introduce a new "C call" protocol, which is an important part of PEP 579. In the reference implementation (which is work in progress), this protocol will be used by built-in functions and methods. However, it should be used by more classes in the future. You find the texts at https://www.python.org/dev/peps/pep-0579 https://www.python.org/dev/peps/pep-0580 Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C-level calling
On 2018-06-20 08:00, Stefan Behnel wrote: Just to add another bit of background on top of the current discussion, there is an idea around, especially in the scipy/big-data community, (and I'm not giving any guarantees here that it will lead to a PEP + implementation, as it depends on people's workload) to design a dedicated C level calling interface for Python. Think of it as similar to the buffer interface, but for calling arbitrary C functions by bypassing the Python call interface entirely. Objects that wrap some kind of C function (and there are tons of them in the CPython world) would gain C signature meta data, maybe even for overloaded signatures, and C code that wants to call them could validate that meta data and call them as native C calls. See also https://www.python.org/dev/peps/pep-0579/#allowing-native-c-arguments I specifically designed PEP 580 to be extendable such that it would be possible to add features later. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
On 19 June 2018 at 16:12, INADA Naoki wrote: > > On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer wrote: >> >> On 2018-06-18 16:55, INADA Naoki wrote: >> > Speeding up most python function and some bultin functions was very >> > significant. >> > But I doubt making some 3rd party call 20% faster can make real >> > applications significant faster. >> >> These two sentences are almost contradictory. I find it strange to claim >> that a given optimization was "very significant" in specific cases while >> saying that the same optimization won't matter in other cases. > > It's not contradictory because there is basis: > > In most real world Python application, number of calling Python methods or > bulitin functions are much more than other calls. > > For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL was > rejected because it's implementation is complex and it's performance gain is > not significant enough on macro benchmarks. > > And I doubt number of 3rd party calls are much more than calling builtin > tp_init or tp_new. I was going to ask a question here about JSON parsing micro-benchmarks, but then I went back and re-read https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html and realised that the main problem discussed in that article was the *memory* overhead of creating full Python object instances, not the runtime cost of instantiating those objects. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods
Hi Jeroen,
On Wed, 20 Jun 2018 10:53:18 +0200
Jeroen Demeyer wrote:
>
> PEP 579 is an informational meta-PEP, listing some of the issues with
> functions/methods implemented in C. The idea is to create several PEPs
> each fix some part of the issues mentioned in PEP 579.
>
> PEP 580 is a standards track PEP to introduce a new "C call" protocol,
> which is an important part of PEP 579. In the reference implementation
> (which is work in progress), this protocol will be used by built-in
> functions and methods. However, it should be used by more classes in the
> future.
This is very detailed and analytic. Thanks.
I dislike that the protocol is complicated, with many special cases.
Ideally there would be two axes for parametrization of C calls:
- the signature of the C callee (either fast call or normal call)
- whether the callable is called as a function ("foo(...)") or as a
method ("some_obj.foo(...)").
But there seems to be some complication on top of that:
- PyCCall_FastCall() accepts several types for the keywords, even a
dict; does it get forwarded as-is to the `cc_func` or is it first
transformed?
- there's CCALL_OBJCLASS and CCALL_SLICE_SELF which have, well,
non-obvious behaviour (especially the latter), especially as it is
conditioned on the value of other fields or flags
I wonder if there's a way to push some of the specificities out of the
protocol and into the C API that mediates between the protocol and
actual callers?
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods
On 2018-06-20 16:09, Antoine Pitrou wrote: But there seems to be some complication on top of that: - PyCCall_FastCall() accepts several types for the keywords, even a dict; That is actually a *simplification* instead of a *complication*. Currently, there is a huge amount of code duplication between _PyMethodDef_RawFastCallKeywords and _PyMethodDef_RawFastCallDict. Folding both of these in one function actually makes things simpler. does it get forwarded as-is to the `cc_func` or is it first transformed? Transformed (obviously, otherwise it would be a huge backwards incompatibility problem). - there's CCALL_OBJCLASS and CCALL_SLICE_SELF which have, well, non-obvious behaviour (especially the latter), especially as it is conditioned on the value of other fields or flags It's actually quite obvious when you think of it: both are needed to support existing use cases. Perhaps it's just not explained well enough in the PEP. I wonder if there's a way to push some of the specificities out of the protocol and into the C API that mediates between the protocol and actual callers? Sorry, I have no idea what you mean here. Actually, those flags are handled by the C API. The actual C functions don't need to care about those flags. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods
On Wed, 20 Jun 2018 16:32:09 +0200 Jeroen Demeyer wrote: > > > - there's CCALL_OBJCLASS and CCALL_SLICE_SELF which have, well, > >non-obvious behaviour (especially the latter), especially as it is > >conditioned on the value of other fields or flags > > It's actually quite obvious when you think of it: both are needed to > support existing use cases. Perhaps it's just not explained well enough > in the PEP. Yes, it's explained in PEP 579. But just because the motivation is easy to understand doesn't mean the mechanism is easy to follow. I'm wondering what amount of code and debugging is needed for, say, Cython or Numba to implement that protocol as a caller, without going through the C API's indirections (for performance). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods
On 2018-06-20 16:42, Antoine Pitrou wrote: I'm wondering what amount of code and debugging is needed for, say, Cython or Numba to implement that protocol as a caller, without going through the C API's indirections (for performance). The goal is to have a really fast C API without a lot of indirections. If Cython or Numba can implement the protocol faster than CPython, we should just change the CPython implementation to be equally fast. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
Hi, All. First of all, thank you Jeroen for writing nice PEPs. When I read PEP 579, I think "6. METH_FASTCALL is private and undocumented" should be solved first. I don't have any idea about changing METH_FASTCALL more. If Victor and Serhiy think so, and PyPy maintainers like it too, I want to make it public as soon as possible. _PyObject_FastCall* APIs are private in Python 3.7. But METH_FASTCALL is not completely private (start without underscore, but not documented) Can we call it as public, stable by adding document, if Ned allows? It's used widely in Python internals already. I suppose that making it public doesn't make Python 3.7 unstable much. If we can't at Python 3.7, I think we should do it at 3.8. Regards, -- INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
Hi, I chose to make it private because I wasn't sure about the API. I was right: Serhiy removed keyword arguments from METH_FASTCALL, you now have to use METH_FASTCALL | METH_KEYWORDS to also pass keyword arguments ;-) I don't recall if this change was done in 3.7 or also in 3.6. FASTCALL has been introduced in 3.6 if I recall correctly. I didn't write much documentation about FASTCALL, only 2 articles. The most interesting one: https://vstinner.github.io/fastcall-microbenchmarks.html METH_FASTCALL is already used by Cython, so I'm not sure that it's fully private :-) > _PyObject_FastCall* APIs are private in Python 3.7. I'm not sure that it's worth it to make these functions public, they are already used internally, when using PyObject_CallFunction() for example. And it may be painful for PyPy (and others) to have to explain all these new functions. > If we can't at Python 3.7, I think we should do it at 3.8. What's the rationale to make it public in 3.7? Can't it wait for 3.8? The new PEPs target 3.8 anyway, no? IMHO it's too late for 3.7. Victor 2018-06-20 17:42 GMT+02:00 INADA Naoki : > Hi, All. > > First of all, thank you Jeroen for writing nice PEPs. > > When I read PEP 579, I think "6. METH_FASTCALL is private and undocumented" > should be solved first. > > I don't have any idea about changing METH_FASTCALL more. > If Victor and Serhiy think so, and PyPy maintainers like it too, I want to > make it public > as soon as possible. > > _PyObject_FastCall* APIs are private in Python 3.7. > But METH_FASTCALL is not completely private (start without underscore, > but not documented) > Can we call it as public, stable by adding document, if Ned allows? > > It's used widely in Python internals already. I suppose that making it > public > doesn't make Python 3.7 unstable much. > > If we can't at Python 3.7, I think we should do it at 3.8. > > Regards, > -- > INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
On Wed, 20 Jun 2018 18:09:00 +0200 Victor Stinner wrote: > > > If we can't at Python 3.7, I think we should do it at 3.8. > > What's the rationale to make it public in 3.7? Can't it wait for 3.8? > The new PEPs target 3.8 anyway, no? > > IMHO it's too late for 3.7. Agreed with Victor. Also Jeroen's work might lead us to change the protocol for better flexibility or performance. Let's not make it a public API too early. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
2018年6月21日(木) 1:17 Antoine Pitrou : > On Wed, 20 Jun 2018 18:09:00 +0200 > Victor Stinner wrote: > > > > > If we can't at Python 3.7, I think we should do it at 3.8. > > > > What's the rationale to make it public in 3.7? Can't it wait for 3.8? > > The new PEPs target 3.8 anyway, no? > > > > IMHO it's too late for 3.7. > > Agreed with Victor. Also Jeroen's work might lead us to change the > protocol for better flexibility or performance. Unless libraries are written with METH_FASTCALL (or using Cython), tp_ccall can't have any gain for 3rd party functions written in C. In other words, if many libraries start supporting FASTCALL, tp_ccall will have more gain at the time when Python 3.8 is released. Let's not make it a > public API too early. > Ok. Even though it's private at 3.7, extension authors can start using it at their risk if we decide METH_FASTCALL is public in 3.8 without any change from 3.7. > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
20.06.18 18:42, INADA Naoki пише: First of all, thank you Jeroen for writing nice PEPs. When I read PEP 579, I think "6. METH_FASTCALL is private and undocumented" should be solved first. I don't have any idea about changing METH_FASTCALL more. If Victor and Serhiy think so, and PyPy maintainers like it too, I want to make it public as soon as possible. I don't have objections against making the METH_FASTCALL method calling convention public. But only for positional-only parameters, the protocol for keyword parameters is more complex and still can be changed. We should to provide also APIs for calling functions using this protocol (_PyObject_FastCall) and for parsing arguments (_PyArg_ParseStack). We may want to bikeshed names and the order of arguments for them. It's used widely in Python internals already. I suppose that making it public doesn't make Python 3.7 unstable much. If we can't at Python 3.7, I think we should do it at 3.8. It is too late for 3.7 in any case. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
2018年6月21日(木) 1:59 Serhiy Storchaka : > 20.06.18 18:42, INADA Naoki пише: > > First of all, thank you Jeroen for writing nice PEPs. > > > > When I read PEP 579, I think "6. METH_FASTCALL is private and > undocumented" > > should be solved first. > > > > I don't have any idea about changing METH_FASTCALL more. > > If Victor and Serhiy think so, and PyPy maintainers like it too, I want > > to make it public > > as soon as possible. > > I don't have objections against making the METH_FASTCALL method calling > convention public. But only for positional-only parameters, the protocol > for keyword parameters is more complex and still can be changed. > > We should to provide also APIs for calling functions using this protocol > (_PyObject_FastCall) and for parsing arguments (_PyArg_ParseStack). We > may want to bikeshed names and the order of arguments for them. > Calling API can be determined later. Even without the API, methods can be called faster from Python core. But for parsing API, you're right. It should be public with METH_FASTCALL. Only positional arguments can be received without it. > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
Serhiy Storchaka schrieb am 20.06.2018 um 18:56: > 20.06.18 18:42, INADA Naoki пише: >> I don't have any idea about changing METH_FASTCALL more. >> If Victor and Serhiy think so, and PyPy maintainers like it too, I want >> to make it public as soon as possible. > > I don't have objections against making the METH_FASTCALL method calling > convention public. But only for positional-only parameters, the protocol > for keyword parameters is more complex and still can be changed. That's also the level that Cython currently uses/supports, exactly because keyword arguments are a) quite a bit more complex, b) a lot less often used and c) pretty much never used in performance critical code. Cython also currently limits the usage to Py3.6+, although I'm considering to generally enable it for everything since Py2.6 as soon as Cython starts using the calling convention for its own functions, just in case it ends up calling itself without prior notice. :) Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
On Wed, 20 Jun 2018 at 09:49 INADA Naoki wrote: > > 2018年6月21日(木) 1:17 Antoine Pitrou : > >> On Wed, 20 Jun 2018 18:09:00 +0200 >> Victor Stinner wrote: >> > >> > > If we can't at Python 3.7, I think we should do it at 3.8. >> > >> > What's the rationale to make it public in 3.7? Can't it wait for 3.8? >> > The new PEPs target 3.8 anyway, no? >> > >> > IMHO it's too late for 3.7. >> >> Agreed with Victor. Also Jeroen's work might lead us to change the >> protocol for better flexibility or performance. > > > Unless libraries are written with METH_FASTCALL (or using Cython), > tp_ccall can't have any gain for 3rd party functions written in C. > > In other words, if many libraries start supporting FASTCALL, tp_ccall will > have more gain at the time when Python 3.8 is released. > > Let's not make it a >> public API too early. >> > > Ok. > > Even though it's private at 3.7, extension authors can start using it at > their risk if we decide METH_FASTCALL is public in 3.8 without any change > from 3.7. > People can still wait for 3.8. Waiting 1.5 years for a feature is nothing when the software you're talking about is already 28 years. :) It's simply not worth the risk. Or you can push for Lukasz's idea of doing annual releases and speed it up a little. ;) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Intent to accept PEP 561 -- Distributing and Packaging Type Information
I have reviewed PEP 561 and I intend to accept it some time next week, unless significant discussion happens between now and then. The latest version of the PEP can be found at https://www.python.org/dev/peps/pep-0561/ PEP 561 solves a big problem for users of static type checkers like mypy and Pyre (as well as pytype and PyCharm): how to scale the creation of stubs (files with just type annotations, with a .pyi extension). IMO Ethan Smith has done a great job both coming up with and revising the design, and crafting an implementation -- most of PEP 561 is already supported by mypy. It's been a while since a copy of the PEP was posted to python-dev ( https://mail.python.org/pipermail/python-dev/2018-April/152694.html), but only a few things changed since then, so I'll just link to the change history: https://github.com/python/peps/commits/master/pep-0561.rst. Only the last two commits are new since the last posting: support for partial packages and a bunch of small textual tweaks I found today while reviewing. There wasn't a lot of feedback then so I don't expect a flamewar today, but better late than never. ;-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
> > >> Even though it's private at 3.7, extension authors can start using it at >> their risk if we decide METH_FASTCALL is public in 3.8 without any change >> from 3.7. >> > > People can still wait for 3.8. Waiting 1.5 years for a feature is nothing > when the software you're talking about is already 28 years. :) It's simply > not worth the risk. > > Of course. My idea is providing information to "early adaptors" who writes C extension manually. PEP 580 is trying to expand METH_FASTCALL to custom function types in 3rd party library written in tools like Cython. But METH_FASTCALL cannot be used widely even for normal function types in 3rd party library yet. Without publicating METH_FASTCALL, PEP 580 is useful only for libraries using private APIs. That's unhealthy. So I think we should discuss about METH_FASTCALL publication before evaluating PEP 580. That's my main point, and "from 3.7" part is just a bonus, sorry. -- INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579
On 2018-06-20 17:42, INADA Naoki wrote: I don't have any idea about changing METH_FASTCALL more. If Victor and Serhiy think so, and PyPy maintainers like it too, I want to make it public as soon as possible. There are two different things here: The first is documenting METH_FASTCALL such that everybody can create built-in functions using the METH_FASTCALL signature. I think that the API for METH_FASTCALL (without or with METH_KEYWORDS) is fine, so I support making it public. This is really just a documentation issue, so I see no reason why it couldn't be added to 3.7.0 if we're fast. The API for calling functions using the FASTCALL convention is more of a mess though. There are functions taking keyword arguments as dict and functions taking them as tuple. As I mentioned in PEP 580, I'd like to merge these and simply allow either a dict or a tuple. Since this would require an API change, this won't be for 3.7.0. Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
