Re: [Cython] CEP 1001 - Custom PyTypeObject extensions
On 05/14/2012 08:01 PM, Robert Bradshaw wrote: On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith wrote: On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn wrote: On 05/14/2012 01:34 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 13.05.2012 21:37: Anyway, thanks for the heads up, this seems to need a bit more work. Input from somebody more familiar with this corner of the CPython API very welcome. Wouldn't you consider python-dev an appropriate place to discuss this? Propose something for a PEP that's primarily useful to Cython without even understanding the full implications myself first? I'd rather try to not annoy people; I figured the time I have the CPython patches ready and tested is the time I ping python-dev... If you want to eventually propose a PEP, you really really really should be talking to them before. Otherwise you'll get everything worked out just the way you want and they'll be like "what is this? re-do it all totally differently". And they might be wrong, but then you have to reconstruct for them the whole debate and reasoning process and implicit assumptions that you're making and not realizing you need to articulate, so easier to just get all the interested people at the table to begin with. And they might be right, in which case you just wasted however much time digging yourself into a hole and reverse-engineering bits of CPython. Don't propose it as a PEP, just say "hey, we have this problem and these constraints, and we're thinking we could solve them by something like this; but of course that has these limitations, so I dunno. What do you think?" And expect to spend some time figuring out what your requirements actually are (even if you think you know already, see above about implicit assumptions). I personally think it's a great idea to bounce ideas around here first before going to python-dev, especially as a PEP wouldn't get in until 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near term. That doesn't preclude presenting the problem and proposed solution on python-dev as well, but the purpose of this thread seems to be to think about it some, including how we're going to support things in the short term, not nail down an exact PEP for Python to accept at face value. I think we're at a point we can ping python-dev now though. To be more futureproof, we'd want an offset to PyExtendedTypeObject rather than assuming it exists at the end of PyTypeObject, but I don't see a good place to store this information, so assuming it's right there based on a bit in the flag seems a reasonable way forward. So I posted on python-dev. There's a lot of "You don't need to do this"; but here's an idea I got that's inspired by that discussion: We could use tp_getattr (and call it directly), but pass in an interned char* which Python code can never get hold of, and then that could return a void* (casted through a PyObject*, but it would not be). Another alternative is to somehow handshake on a metaclass implementation; and different Cython modules/NumPy/SciPy etc. would inherit from it. But apart from that handshaking, and having to use a metaclass and make everything more complicated for C implementors of the spec, it gives you a more expensive check than just checking a flag. I like a flag bit much better. I still hope that somebody more understanding comes along, argues our case, and gets bit 22 reserved for our purpose :-) Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CEP 1001 - Custom PyTypeObject extensions
On 16 May 2012 12:20, Dag Sverre Seljebotn wrote: > On 05/14/2012 08:01 PM, Robert Bradshaw wrote: >> >> On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith wrote: >>> >>> On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn >>> wrote: On 05/14/2012 01:34 PM, Stefan Behnel wrote: > > > Dag Sverre Seljebotn, 13.05.2012 21:37: >> >> >> Anyway, thanks for the heads up, this seems to need a bit more work. >> Input >> from somebody more familiar with this corner of the CPython API very >> welcome. > > > > Wouldn't you consider python-dev an appropriate place to discuss this? Propose something for a PEP that's primarily useful to Cython without even understanding the full implications myself first? I'd rather try to not annoy people; I figured the time I have the CPython patches ready and tested is the time I ping python-dev... >>> >>> >>> If you want to eventually propose a PEP, you really really really >>> should be talking to them before. Otherwise you'll get everything >>> worked out just the way you want and they'll be like "what is this? >>> re-do it all totally differently". And they might be wrong, but then >>> you have to reconstruct for them the whole debate and reasoning >>> process and implicit assumptions that you're making and not realizing >>> you need to articulate, so easier to just get all the interested >>> people at the table to begin with. And they might be right, in which >>> case you just wasted however much time digging yourself into a hole >>> and reverse-engineering bits of CPython. >>> >>> Don't propose it as a PEP, just say "hey, we have this problem and >>> these constraints, and we're thinking we could solve them by something >>> like this; but of course that has these limitations, so I dunno. What >>> do you think?" And expect to spend some time figuring out what your >>> requirements actually are (even if you think you know already, see >>> above about implicit assumptions). >> >> >> I personally think it's a great idea to bounce ideas around here first >> before going to python-dev, especially as a PEP wouldn't get in until >> 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near >> term. That doesn't preclude presenting the problem and proposed >> solution on python-dev as well, but the purpose of this thread seems >> to be to think about it some, including how we're going to support >> things in the short term, not nail down an exact PEP for Python to >> accept at face value. I think we're at a point we can ping python-dev >> now though. >> >> To be more futureproof, we'd want an offset to PyExtendedTypeObject >> rather than assuming it exists at the end of PyTypeObject, but I don't >> see a good place to store this information, so assuming it's right >> there based on a bit in the flag seems a reasonable way forward. > > > So I posted on python-dev. > > There's a lot of "You don't need to do this"; but here's an idea I got > that's inspired by that discussion: We could use tp_getattr (and call it > directly), but pass in an interned char* which Python code can never get > hold of, and then that could return a void* (casted through a PyObject*, but > it would not be). Would we want to support monkey patching these interfaces? If so, this mechanism would be a bit harder than reallocing a pointer, although I guess a closure chain of tp_getattr functions would work :). But I think we want GIL-less access anyway right, which means neither approach would work unsynchronized. > Another alternative is to somehow handshake on a metaclass implementation; > and different Cython modules/NumPy/SciPy etc. would inherit from it. But > apart from that handshaking, and having to use a metaclass and make > everything more complicated for C implementors of the spec, it gives you a > more expensive check than just checking a flag. I agree that the flag is much easier, if you have a metaclass the question is again in which module to store it to get a cross-module working typecheck. On the other hand, if the header file provides an easy way to import the metaclass (monkeypatched on some module or living in its own module), and to allocate type instances given a (statically allocated) type, that would be more future-proof and elegant. I don't think it would be much slower, it's doing 'o->ob_type->tp_flags & MYFLAG' vs 'o->ob_type->ob_type == MYMETA'. I think for the bit flag the interface won't span subclasses, whereas the metaclass approach would allow subclassing but not subclassing of the metaclass itself (unless you instantiate the metaclass through itself, and check the interface against the metaclass, which only means the metaclass of the metaclass isn't subclassable :) (this would also mean a more expensive check)). I think if we implement CEP 1000, we will at that time have a generic way to optimize and hoist none checking/bounds checking etc, which will als
Re: [Cython] CEP 1001 - Custom PyTypeObject extensions
On 16 May 2012 14:03, mark florisson wrote: > On 16 May 2012 12:20, Dag Sverre Seljebotn wrote: >> On 05/14/2012 08:01 PM, Robert Bradshaw wrote: >>> >>> On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith wrote: On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn wrote: > > On 05/14/2012 01:34 PM, Stefan Behnel wrote: >> >> >> Dag Sverre Seljebotn, 13.05.2012 21:37: >>> >>> >>> Anyway, thanks for the heads up, this seems to need a bit more work. >>> Input >>> from somebody more familiar with this corner of the CPython API very >>> welcome. >> >> >> >> Wouldn't you consider python-dev an appropriate place to discuss this? > > > > Propose something for a PEP that's primarily useful to Cython without > even > understanding the full implications myself first? > > I'd rather try to not annoy people; I figured the time I have the > CPython > patches ready and tested is the time I ping python-dev... If you want to eventually propose a PEP, you really really really should be talking to them before. Otherwise you'll get everything worked out just the way you want and they'll be like "what is this? re-do it all totally differently". And they might be wrong, but then you have to reconstruct for them the whole debate and reasoning process and implicit assumptions that you're making and not realizing you need to articulate, so easier to just get all the interested people at the table to begin with. And they might be right, in which case you just wasted however much time digging yourself into a hole and reverse-engineering bits of CPython. Don't propose it as a PEP, just say "hey, we have this problem and these constraints, and we're thinking we could solve them by something like this; but of course that has these limitations, so I dunno. What do you think?" And expect to spend some time figuring out what your requirements actually are (even if you think you know already, see above about implicit assumptions). >>> >>> >>> I personally think it's a great idea to bounce ideas around here first >>> before going to python-dev, especially as a PEP wouldn't get in until >>> 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near >>> term. That doesn't preclude presenting the problem and proposed >>> solution on python-dev as well, but the purpose of this thread seems >>> to be to think about it some, including how we're going to support >>> things in the short term, not nail down an exact PEP for Python to >>> accept at face value. I think we're at a point we can ping python-dev >>> now though. >>> >>> To be more futureproof, we'd want an offset to PyExtendedTypeObject >>> rather than assuming it exists at the end of PyTypeObject, but I don't >>> see a good place to store this information, so assuming it's right >>> there based on a bit in the flag seems a reasonable way forward. >> >> >> So I posted on python-dev. >> >> There's a lot of "You don't need to do this"; but here's an idea I got >> that's inspired by that discussion: We could use tp_getattr (and call it >> directly), but pass in an interned char* which Python code can never get >> hold of, and then that could return a void* (casted through a PyObject*, but >> it would not be). > > Would we want to support monkey patching these interfaces? If so, this > mechanism would be a bit harder than reallocing a pointer, although I > guess a closure chain of tp_getattr functions would work :). But I > think we want GIL-less access anyway right, which means neither > approach would work unsynchronized. > >> Another alternative is to somehow handshake on a metaclass implementation; >> and different Cython modules/NumPy/SciPy etc. would inherit from it. But >> apart from that handshaking, and having to use a metaclass and make >> everything more complicated for C implementors of the spec, it gives you a >> more expensive check than just checking a flag. > > I agree that the flag is much easier, if you have a metaclass the > question is again in which module to store it to get a cross-module > working typecheck. On the other hand, if the header file provides an > easy way to import the metaclass (monkeypatched on some module or > living in its own module), and to allocate type instances given a > (statically allocated) type, that would be more future-proof and > elegant. I don't think it would be much slower, it's doing > 'o->ob_type->tp_flags & MYFLAG' vs 'o->ob_type->ob_type == MYMETA'. Sorry, I think you mentioned something in related in the CEP, I couldn't find it from the enhancement list (I should have followed your originally posted link :). So that's a good point, there are several issues: - subclasses of the type exposing the interface (I don't think that can be handled through tp_flags) - in the case of a metaclass approach, subclass
Re: [Cython] CEP 1001 - Custom PyTypeObject extensions
On 16 May 2012 14:25, mark florisson wrote: > On 16 May 2012 14:03, mark florisson wrote: >> On 16 May 2012 12:20, Dag Sverre Seljebotn >> wrote: >>> On 05/14/2012 08:01 PM, Robert Bradshaw wrote: On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith wrote: > > On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn > wrote: >> >> On 05/14/2012 01:34 PM, Stefan Behnel wrote: >>> >>> >>> Dag Sverre Seljebotn, 13.05.2012 21:37: Anyway, thanks for the heads up, this seems to need a bit more work. Input from somebody more familiar with this corner of the CPython API very welcome. >>> >>> >>> >>> Wouldn't you consider python-dev an appropriate place to discuss this? >> >> >> >> Propose something for a PEP that's primarily useful to Cython without >> even >> understanding the full implications myself first? >> >> I'd rather try to not annoy people; I figured the time I have the >> CPython >> patches ready and tested is the time I ping python-dev... > > > If you want to eventually propose a PEP, you really really really > should be talking to them before. Otherwise you'll get everything > worked out just the way you want and they'll be like "what is this? > re-do it all totally differently". And they might be wrong, but then > you have to reconstruct for them the whole debate and reasoning > process and implicit assumptions that you're making and not realizing > you need to articulate, so easier to just get all the interested > people at the table to begin with. And they might be right, in which > case you just wasted however much time digging yourself into a hole > and reverse-engineering bits of CPython. > > Don't propose it as a PEP, just say "hey, we have this problem and > these constraints, and we're thinking we could solve them by something > like this; but of course that has these limitations, so I dunno. What > do you think?" And expect to spend some time figuring out what your > requirements actually are (even if you think you know already, see > above about implicit assumptions). I personally think it's a great idea to bounce ideas around here first before going to python-dev, especially as a PEP wouldn't get in until 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near term. That doesn't preclude presenting the problem and proposed solution on python-dev as well, but the purpose of this thread seems to be to think about it some, including how we're going to support things in the short term, not nail down an exact PEP for Python to accept at face value. I think we're at a point we can ping python-dev now though. To be more futureproof, we'd want an offset to PyExtendedTypeObject rather than assuming it exists at the end of PyTypeObject, but I don't see a good place to store this information, so assuming it's right there based on a bit in the flag seems a reasonable way forward. >>> >>> >>> So I posted on python-dev. >>> >>> There's a lot of "You don't need to do this"; but here's an idea I got >>> that's inspired by that discussion: We could use tp_getattr (and call it >>> directly), but pass in an interned char* which Python code can never get >>> hold of, and then that could return a void* (casted through a PyObject*, but >>> it would not be). >> >> Would we want to support monkey patching these interfaces? If so, this >> mechanism would be a bit harder than reallocing a pointer, although I >> guess a closure chain of tp_getattr functions would work :). But I >> think we want GIL-less access anyway right, which means neither >> approach would work unsynchronized. >> >>> Another alternative is to somehow handshake on a metaclass implementation; >>> and different Cython modules/NumPy/SciPy etc. would inherit from it. But >>> apart from that handshaking, and having to use a metaclass and make >>> everything more complicated for C implementors of the spec, it gives you a >>> more expensive check than just checking a flag. >> >> I agree that the flag is much easier, if you have a metaclass the >> question is again in which module to store it to get a cross-module >> working typecheck. On the other hand, if the header file provides an >> easy way to import the metaclass (monkeypatched on some module or >> living in its own module), and to allocate type instances given a >> (statically allocated) type, that would be more future-proof and >> elegant. I don't think it would be much slower, it's doing >> 'o->ob_type->tp_flags & MYFLAG' vs 'o->ob_type->ob_type == MYMETA'. > > Sorry, I think you mentioned something in related in the CEP, I > couldn't find it from the enhancement list (I should have followed > your originally posted link :). So that's a good point, there are > several issues:
Re: [Cython] CEP 1001 - Custom PyTypeObject extensions
On 16 May 2012 14:25, mark florisson wrote: > On 16 May 2012 14:03, mark florisson wrote: >> On 16 May 2012 12:20, Dag Sverre Seljebotn >> wrote: >>> On 05/14/2012 08:01 PM, Robert Bradshaw wrote: On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith wrote: > > On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn > wrote: >> >> On 05/14/2012 01:34 PM, Stefan Behnel wrote: >>> >>> >>> Dag Sverre Seljebotn, 13.05.2012 21:37: Anyway, thanks for the heads up, this seems to need a bit more work. Input from somebody more familiar with this corner of the CPython API very welcome. >>> >>> >>> >>> Wouldn't you consider python-dev an appropriate place to discuss this? >> >> >> >> Propose something for a PEP that's primarily useful to Cython without >> even >> understanding the full implications myself first? >> >> I'd rather try to not annoy people; I figured the time I have the >> CPython >> patches ready and tested is the time I ping python-dev... > > > If you want to eventually propose a PEP, you really really really > should be talking to them before. Otherwise you'll get everything > worked out just the way you want and they'll be like "what is this? > re-do it all totally differently". And they might be wrong, but then > you have to reconstruct for them the whole debate and reasoning > process and implicit assumptions that you're making and not realizing > you need to articulate, so easier to just get all the interested > people at the table to begin with. And they might be right, in which > case you just wasted however much time digging yourself into a hole > and reverse-engineering bits of CPython. > > Don't propose it as a PEP, just say "hey, we have this problem and > these constraints, and we're thinking we could solve them by something > like this; but of course that has these limitations, so I dunno. What > do you think?" And expect to spend some time figuring out what your > requirements actually are (even if you think you know already, see > above about implicit assumptions). I personally think it's a great idea to bounce ideas around here first before going to python-dev, especially as a PEP wouldn't get in until 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near term. That doesn't preclude presenting the problem and proposed solution on python-dev as well, but the purpose of this thread seems to be to think about it some, including how we're going to support things in the short term, not nail down an exact PEP for Python to accept at face value. I think we're at a point we can ping python-dev now though. To be more futureproof, we'd want an offset to PyExtendedTypeObject rather than assuming it exists at the end of PyTypeObject, but I don't see a good place to store this information, so assuming it's right there based on a bit in the flag seems a reasonable way forward. >>> >>> >>> So I posted on python-dev. >>> >>> There's a lot of "You don't need to do this"; but here's an idea I got >>> that's inspired by that discussion: We could use tp_getattr (and call it >>> directly), but pass in an interned char* which Python code can never get >>> hold of, and then that could return a void* (casted through a PyObject*, but >>> it would not be). >> >> Would we want to support monkey patching these interfaces? If so, this >> mechanism would be a bit harder than reallocing a pointer, although I >> guess a closure chain of tp_getattr functions would work :). But I >> think we want GIL-less access anyway right, which means neither >> approach would work unsynchronized. >> >>> Another alternative is to somehow handshake on a metaclass implementation; >>> and different Cython modules/NumPy/SciPy etc. would inherit from it. But >>> apart from that handshaking, and having to use a metaclass and make >>> everything more complicated for C implementors of the spec, it gives you a >>> more expensive check than just checking a flag. >> >> I agree that the flag is much easier, if you have a metaclass the >> question is again in which module to store it to get a cross-module >> working typecheck. On the other hand, if the header file provides an >> easy way to import the metaclass (monkeypatched on some module or >> living in its own module), and to allocate type instances given a >> (statically allocated) type, that would be more future-proof and >> elegant. I don't think it would be much slower, it's doing >> 'o->ob_type->tp_flags & MYFLAG' vs 'o->ob_type->ob_type == MYMETA'. > > Sorry, I think you mentioned something in related in the CEP, I > couldn't find it from the enhancement list (I should have followed > your originally posted link :). So that's a good point, there are > several issues:
Re: [Cython] [cython] Python array support (#113)
Andreas van Cranenburgh, 16.05.2012 18:15: > Any news on this? Let me know if there's anything I can do to help inclusion > of this patch. Could someone please take over here? https://github.com/cython/cython/pull/113 I haven't merged this yet and won't have the time to do it soonish. What I'd like to see happen is to get the current header file replaced by utility code "somehow". Not sure how that "somehow" is going to work. Basically, if this can be solved, I'd love to have it in for 0.17. Otherwise, well, not ... Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [Python-Dev] C-level duck typing
"Martin v. Löwis", 16.05.2012 20:33: >> Does this use case make sense to everyone? >> >> The reason why we are discussing this on python-dev is that we are looking >> for a general way to expose these C level signatures within the Python >> ecosystem. And Dag's idea was to expose them as part of the type object, >> basically as an addition to the current Python level tp_call() slot. > > The use case makes sense, yet there is also a long-standing solution > already to expose APIs and function pointers: the capsule objects. > > If you want to avoid dictionary lookups on the server side, implement > tp_getattro, comparing addresses of interned strings. I think Martin has a point there. Why not just use a custom attribute on callables that hold a PyCapsule? Whenever we see inside of a Cython implemented function that an object variable that was retrieved from the outside, either as a function argument or as the result of a function call, is being called, we try to unpack a C function pointer from it on all assignments to the variable. If that works, we can scan for a suitable signature (either right away or lazily on first access) and cache that. On each subsequent call through that variable, the cached C function will be used. That means we'd replace Python variables that are being called by multiple local variables, one that holds the object and one for each C function with a different signature that it is being called with. We set the C function variables to NULL when the Python function variable is being assigned to. When the C function variable is NULL on call, we scan for a matching signature and assign it to the variable. When no matching signature can be found, we set it to (void*)-1. Additionally, we allow explicit user casts of Python objects to C function types, which would then try to unpack the C function, raising a TypeError on mismatch. Assignments to callable variables can be expected to occur much less frequently than calls to them, so this will give us a good trade-off in most cases. I don't see why this kind of caching would be any slower inside of loops than what we were discussing so far. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [Python-Dev] C-level duck typing
On 16 May 2012 20:15, Stefan Behnel wrote: > "Martin v. Löwis", 16.05.2012 20:33: >>> Does this use case make sense to everyone? >>> >>> The reason why we are discussing this on python-dev is that we are looking >>> for a general way to expose these C level signatures within the Python >>> ecosystem. And Dag's idea was to expose them as part of the type object, >>> basically as an addition to the current Python level tp_call() slot. >> >> The use case makes sense, yet there is also a long-standing solution >> already to expose APIs and function pointers: the capsule objects. >> >> If you want to avoid dictionary lookups on the server side, implement >> tp_getattro, comparing addresses of interned strings. > > I think Martin has a point there. Why not just use a custom attribute on > callables that hold a PyCapsule? Whenever we see inside of a Cython > implemented function that an object variable that was retrieved from the > outside, either as a function argument or as the result of a function call, > is being called, we try to unpack a C function pointer from it on all > assignments to the variable. If that works, we can scan for a suitable > signature (either right away or lazily on first access) and cache that. On > each subsequent call through that variable, the cached C function will be > used. > > That means we'd replace Python variables that are being called by multiple > local variables, one that holds the object and one for each C function with > a different signature that it is being called with. We set the C function > variables to NULL when the Python function variable is being assigned to. > When the C function variable is NULL on call, we scan for a matching > signature and assign it to the variable. When no matching signature can be > found, we set it to (void*)-1. > > Additionally, we allow explicit user casts of Python objects to C function > types, which would then try to unpack the C function, raising a TypeError > on mismatch. > > Assignments to callable variables can be expected to occur much less > frequently than calls to them, so this will give us a good trade-off in > most cases. I don't see why this kind of caching would be any slower inside > of loops than what we were discussing so far. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel This works really well for local variables, but for globals, def methods or callbacks as attributes, this won't work so well, as they may be rebound at any time outside of the module scope. I think in general Cython code could be easily sped up for most cases by provided a really fast dispatch mechanism here. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [Python-Dev] C-level duck typing
On 16 May 2012 20:49, mark florisson wrote: > On 16 May 2012 20:15, Stefan Behnel wrote: >> "Martin v. Löwis", 16.05.2012 20:33: Does this use case make sense to everyone? The reason why we are discussing this on python-dev is that we are looking for a general way to expose these C level signatures within the Python ecosystem. And Dag's idea was to expose them as part of the type object, basically as an addition to the current Python level tp_call() slot. >>> >>> The use case makes sense, yet there is also a long-standing solution >>> already to expose APIs and function pointers: the capsule objects. >>> >>> If you want to avoid dictionary lookups on the server side, implement >>> tp_getattro, comparing addresses of interned strings. >> >> I think Martin has a point there. Why not just use a custom attribute on >> callables that hold a PyCapsule? Whenever we see inside of a Cython >> implemented function that an object variable that was retrieved from the >> outside, either as a function argument or as the result of a function call, >> is being called, we try to unpack a C function pointer from it on all >> assignments to the variable. If that works, we can scan for a suitable >> signature (either right away or lazily on first access) and cache that. On >> each subsequent call through that variable, the cached C function will be >> used. >> >> That means we'd replace Python variables that are being called by multiple >> local variables, one that holds the object and one for each C function with >> a different signature that it is being called with. We set the C function >> variables to NULL when the Python function variable is being assigned to. >> When the C function variable is NULL on call, we scan for a matching >> signature and assign it to the variable. When no matching signature can be >> found, we set it to (void*)-1. >> >> Additionally, we allow explicit user casts of Python objects to C function >> types, which would then try to unpack the C function, raising a TypeError >> on mismatch. >> >> Assignments to callable variables can be expected to occur much less >> frequently than calls to them, so this will give us a good trade-off in >> most cases. I don't see why this kind of caching would be any slower inside >> of loops than what we were discussing so far. >> >> Stefan >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > This works really well for local variables, but for globals, def > methods or callbacks as attributes, this won't work so well, as they > may be rebound at any time outside of the module scope. I think in > general Cython code could be easily sped up for most cases by provided > a really fast dispatch mechanism here. ... unless we implement the __nomonkey__ (forgot the original name) or final declaration (also allowed in pxd files to declare module attributes final), where you can declare module attributes or class attributes final. I don't recall the outcome of the discussion, but I suppose the advantage of the __nomonkey__ is that is works from Python code as well and you don't have to bother with boring pxds, whereas the advantage of final is that is can work for class attributes. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [Python-Dev] C-level duck typing
On Wed, 16 May 2012 20:49:18 +0100, mark florisson wrote: On 16 May 2012 20:15, Stefan Behnel wrote: "Martin v. Löwis", 16.05.2012 20:33: Does this use case make sense to everyone? The reason why we are discussing this on python-dev is that we are looking for a general way to expose these C level signatures within the Python ecosystem. And Dag's idea was to expose them as part of the type object, basically as an addition to the current Python level tp_call() slot. The use case makes sense, yet there is also a long-standing solution already to expose APIs and function pointers: the capsule objects. If you want to avoid dictionary lookups on the server side, implement tp_getattro, comparing addresses of interned strings. I think Martin has a point there. Why not just use a custom attribute on callables that hold a PyCapsule? Whenever we see inside of a Cython implemented function that an object variable that was retrieved from the outside, either as a function argument or as the result of a function call, is being called, we try to unpack a C function pointer from it on all assignments to the variable. If that works, we can scan for a suitable signature (either right away or lazily on first access) and cache that. On each subsequent call through that variable, the cached C function will be used. That means we'd replace Python variables that are being called by multiple local variables, one that holds the object and one for each C function with a different signature that it is being called with. We set the C function variables to NULL when the Python function variable is being assigned to. When the C function variable is NULL on call, we scan for a matching signature and assign it to the variable. When no matching signature can be found, we set it to (void*)-1. Additionally, we allow explicit user casts of Python objects to C function types, which would then try to unpack the C function, raising a TypeError on mismatch. Assignments to callable variables can be expected to occur much less frequently than calls to them, so this will give us a good trade-off in most cases. I don't see why this kind of caching would be any slower inside of loops than what we were discussing so far. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel This works really well for local variables, but for globals, def methods or callbacks as attributes, this won't work so well, as they may be rebound at any time outside of the module scope. I think in +1. The python-dev discussion is pretty focused on the world of a manually written C extension. But code generation is an entirely different matter. Python puts in place pretty efficient boundaries against full-program static analysis, so there's really not much we can do. Here's some of my actual code I have for wrapping a C++ library: cdef class CallbackEventReceiver(BasicEventReceiver): cdef object callback def __init__(self, callback): self.callback = callback cdef dispatch_event(self, ...): self.callback(...) The idea is that you can subclass BasicEventReceiver in Cython for speed, but if you want to use a Python callable then this converter is used. This code is very performance critical. And, the *loop* in question sits deep inside a C++ library. Good luck pre-acquiring the function pointer of self.callback in any useful way. Even if it is not exported by the class, that could be overridden by a subclass. I stress the fact that this is real world code by yours truly (unfortunately not open source, it wraps a closed source library). Yes, you can tell users to be mindful of this and make as much as possible local variables, introduce final modifiers and __nomonkey__ and whatnot, but that's a large price to pay to avoid hacking tp_flags. Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [Python-Dev] C-level duck typing
On Wed, May 16, 2012 at 12:15 PM, Stefan Behnel wrote: > "Martin v. Löwis", 16.05.2012 20:33: >>> Does this use case make sense to everyone? >>> >>> The reason why we are discussing this on python-dev is that we are looking >>> for a general way to expose these C level signatures within the Python >>> ecosystem. And Dag's idea was to expose them as part of the type object, >>> basically as an addition to the current Python level tp_call() slot. >> >> The use case makes sense, yet there is also a long-standing solution >> already to expose APIs and function pointers: the capsule objects. >> >> If you want to avoid dictionary lookups on the server side, implement >> tp_getattro, comparing addresses of interned strings. > > I think Martin has a point there. Why not just use a custom attribute on > callables that hold a PyCapsule? Whenever we see inside of a Cython > implemented function that an object variable that was retrieved from the > outside, either as a function argument or as the result of a function call, > is being called, we try to unpack a C function pointer from it on all > assignments to the variable. If that works, we can scan for a suitable > signature (either right away or lazily on first access) and cache that. On > each subsequent call through that variable, the cached C function will be > used. > > That means we'd replace Python variables that are being called by multiple > local variables, one that holds the object and one for each C function with > a different signature that it is being called with. We set the C function > variables to NULL when the Python function variable is being assigned to. > When the C function variable is NULL on call, we scan for a matching > signature and assign it to the variable. When no matching signature can be > found, we set it to (void*)-1. > > Additionally, we allow explicit user casts of Python objects to C function > types, which would then try to unpack the C function, raising a TypeError > on mismatch. > > Assignments to callable variables can be expected to occur much less > frequently than calls to them, so this will give us a good trade-off in > most cases. I don't see why this kind of caching would be any slower inside > of loops than what we were discussing so far. I like the idea, but that only helps if you're doing multiple calls (e.g. in a loop). Definitely worth implementing in my mind, but orthogonal to making the lookup itself as fast as possible. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [Python-Dev] C-level duck typing
On 16 May 2012 21:16, Dag Sverre Seljebotn wrote: > On Wed, 16 May 2012 20:49:18 +0100, mark florisson > wrote: >> >> On 16 May 2012 20:15, Stefan Behnel wrote: >>> >>> "Martin v. Löwis", 16.05.2012 20:33: > > Does this use case make sense to everyone? > > The reason why we are discussing this on python-dev is that we are > looking > for a general way to expose these C level signatures within the Python > ecosystem. And Dag's idea was to expose them as part of the type > object, > basically as an addition to the current Python level tp_call() slot. The use case makes sense, yet there is also a long-standing solution already to expose APIs and function pointers: the capsule objects. If you want to avoid dictionary lookups on the server side, implement tp_getattro, comparing addresses of interned strings. >>> >>> >>> I think Martin has a point there. Why not just use a custom attribute on >>> callables that hold a PyCapsule? Whenever we see inside of a Cython >>> implemented function that an object variable that was retrieved from the >>> outside, either as a function argument or as the result of a function >>> call, >>> is being called, we try to unpack a C function pointer from it on all >>> assignments to the variable. If that works, we can scan for a suitable >>> signature (either right away or lazily on first access) and cache that. >>> On >>> each subsequent call through that variable, the cached C function will be >>> used. >>> >>> That means we'd replace Python variables that are being called by >>> multiple >>> local variables, one that holds the object and one for each C function >>> with >>> a different signature that it is being called with. We set the C function >>> variables to NULL when the Python function variable is being assigned to. >>> When the C function variable is NULL on call, we scan for a matching >>> signature and assign it to the variable. When no matching signature can >>> be >>> found, we set it to (void*)-1. >>> >>> Additionally, we allow explicit user casts of Python objects to C >>> function >>> types, which would then try to unpack the C function, raising a TypeError >>> on mismatch. >>> >>> Assignments to callable variables can be expected to occur much less >>> frequently than calls to them, so this will give us a good trade-off in >>> most cases. I don't see why this kind of caching would be any slower >>> inside >>> of loops than what we were discussing so far. >>> >>> Stefan >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >> >> >> This works really well for local variables, but for globals, def >> methods or callbacks as attributes, this won't work so well, as they >> may be rebound at any time outside of the module scope. I think in > > > +1. The python-dev discussion is pretty focused on the world of a manually > written C extension. But code generation is an entirely different matter. > Python puts in place pretty efficient boundaries against full-program static > analysis, so there's really not much we can do. > > Here's some of my actual code I have for wrapping a C++ library: > > cdef class CallbackEventReceiver(BasicEventReceiver): > cdef object callback > > def __init__(self, callback): > self.callback = callback > > cdef dispatch_event(self, ...): > self.callback(...) > > The idea is that you can subclass BasicEventReceiver in Cython for speed, > but if you want to use a Python callable then this converter is used. > > This code is very performance critical. And, the *loop* in question sits > deep inside a C++ library. > > Good luck pre-acquiring the function pointer of self.callback in any useful > way. Even if it is not exported by the class, that could be overridden by a > subclass. I stress the fact that this is real world code by yours truly > (unfortunately not open source, it wraps a closed source library). > > Yes, you can tell users to be mindful of this and make as much as possible > local variables, introduce final modifiers and __nomonkey__ and whatnot, but > that's a large price to pay to avoid hacking tp_flags. > > Dag > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel Definitely. I personally prefer the metaclass approach, but it's an irrelevant detail. If we go the tp_flags route, would we copy all the interface information from the superclass into the subclass directly? I think in any case we need a wrapper around PyType_Ready to inherit the tp_flags bit (which would be automatic with a metaclass). ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [Python-Dev] C-level duck typing
On 16 May 2012 21:16, Dag Sverre Seljebotn wrote: > On Wed, 16 May 2012 20:49:18 +0100, mark florisson > wrote: >> >> On 16 May 2012 20:15, Stefan Behnel wrote: >>> >>> "Martin v. Löwis", 16.05.2012 20:33: > > Does this use case make sense to everyone? > > The reason why we are discussing this on python-dev is that we are > looking > for a general way to expose these C level signatures within the Python > ecosystem. And Dag's idea was to expose them as part of the type > object, > basically as an addition to the current Python level tp_call() slot. The use case makes sense, yet there is also a long-standing solution already to expose APIs and function pointers: the capsule objects. If you want to avoid dictionary lookups on the server side, implement tp_getattro, comparing addresses of interned strings. >>> >>> >>> I think Martin has a point there. Why not just use a custom attribute on >>> callables that hold a PyCapsule? Whenever we see inside of a Cython >>> implemented function that an object variable that was retrieved from the >>> outside, either as a function argument or as the result of a function >>> call, >>> is being called, we try to unpack a C function pointer from it on all >>> assignments to the variable. If that works, we can scan for a suitable >>> signature (either right away or lazily on first access) and cache that. >>> On >>> each subsequent call through that variable, the cached C function will be >>> used. >>> >>> That means we'd replace Python variables that are being called by >>> multiple >>> local variables, one that holds the object and one for each C function >>> with >>> a different signature that it is being called with. We set the C function >>> variables to NULL when the Python function variable is being assigned to. >>> When the C function variable is NULL on call, we scan for a matching >>> signature and assign it to the variable. When no matching signature can >>> be >>> found, we set it to (void*)-1. >>> >>> Additionally, we allow explicit user casts of Python objects to C >>> function >>> types, which would then try to unpack the C function, raising a TypeError >>> on mismatch. >>> >>> Assignments to callable variables can be expected to occur much less >>> frequently than calls to them, so this will give us a good trade-off in >>> most cases. I don't see why this kind of caching would be any slower >>> inside >>> of loops than what we were discussing so far. >>> >>> Stefan >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >> >> >> This works really well for local variables, but for globals, def >> methods or callbacks as attributes, this won't work so well, as they >> may be rebound at any time outside of the module scope. I think in > > > +1. The python-dev discussion is pretty focused on the world of a manually > written C extension. But code generation is an entirely different matter. > Python puts in place pretty efficient boundaries against full-program static > analysis, so there's really not much we can do. > > Here's some of my actual code I have for wrapping a C++ library: > > cdef class CallbackEventReceiver(BasicEventReceiver): > cdef object callback > > def __init__(self, callback): > self.callback = callback > > cdef dispatch_event(self, ...): > self.callback(...) > > The idea is that you can subclass BasicEventReceiver in Cython for speed, > but if you want to use a Python callable then this converter is used. > > This code is very performance critical. And, the *loop* in question sits > deep inside a C++ library. > > Good luck pre-acquiring the function pointer of self.callback in any useful > way. Even if it is not exported by the class, that could be overridden by a > subclass. I stress the fact that this is real world code by yours truly > (unfortunately not open source, it wraps a closed source library). > > Yes, you can tell users to be mindful of this and make as much as possible > local variables, introduce final modifiers and __nomonkey__ and whatnot, but > that's a large price to pay to avoid hacking tp_flags. > > Dag > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel I suppose for this case it might be faster to check if the world is sane (if the callback or function is still the object you expect it to be) on top of looking at whether the function pointer is unpacked. You don't really want to store that extra information in objects, but for global variables it might be worth the while (unless you're doing import * :)). So we definitely always need a fast dispatcher, but we may do slightly better in some cases if we care to implement it. I bet no one will care about shaving off those last 2 nano seconds though :) __
Re: [Cython] [Python-Dev] C-level duck typing
mark florisson, 16.05.2012 21:49: > On 16 May 2012 20:15, Stefan Behnel wrote: >> "Martin v. Löwis", 16.05.2012 20:33: Does this use case make sense to everyone? The reason why we are discussing this on python-dev is that we are looking for a general way to expose these C level signatures within the Python ecosystem. And Dag's idea was to expose them as part of the type object, basically as an addition to the current Python level tp_call() slot. >>> >>> The use case makes sense, yet there is also a long-standing solution >>> already to expose APIs and function pointers: the capsule objects. >>> >>> If you want to avoid dictionary lookups on the server side, implement >>> tp_getattro, comparing addresses of interned strings. >> >> I think Martin has a point there. Why not just use a custom attribute on >> callables that hold a PyCapsule? Whenever we see inside of a Cython >> implemented function that an object variable that was retrieved from the >> outside, either as a function argument or as the result of a function call, >> is being called, we try to unpack a C function pointer from it on all >> assignments to the variable. If that works, we can scan for a suitable >> signature (either right away or lazily on first access) and cache that. On >> each subsequent call through that variable, the cached C function will be >> used. >> >> That means we'd replace Python variables that are being called by multiple >> local variables, one that holds the object and one for each C function with >> a different signature that it is being called with. We set the C function >> variables to NULL when the Python function variable is being assigned to. >> When the C function variable is NULL on call, we scan for a matching >> signature and assign it to the variable. When no matching signature can be >> found, we set it to (void*)-1. >> >> Additionally, we allow explicit user casts of Python objects to C function >> types, which would then try to unpack the C function, raising a TypeError >> on mismatch. >> >> Assignments to callable variables can be expected to occur much less >> frequently than calls to them, so this will give us a good trade-off in >> most cases. I don't see why this kind of caching would be any slower inside >> of loops than what we were discussing so far. > > This works really well for local variables, but for globals, def > methods or callbacks as attributes, this won't work so well, as they > may be rebound at any time outside of the module scope. Only half true for globals, which can be declared "cdef object", e.g. for imported names. That would allow Cython to see all possible reassignments in a module, which would then apply the above scheme. I don't think def methods are a use case for this because you'd either cpdef them or even cdef them if you want speed. If you want them to be overridable, you'll have to live with the speed penalty that that implies. For object attributes, you have to pay the penalty of a lookup anyway, no way around that. We can't even cache anything here (e.g. with a borrowed reference) because the attribute may be rebound to another object that happens to live at the same address as the previous one. However, if you want speed, you'd do it as in CPython and assign the object to a local variable to pay the lookup of only once. Problem solved. > I think in > general Cython code could be easily sped up for most cases by provided > a really fast dispatch mechanism here. I feel inclined to doubt that by now. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
