[Cython] Mitigating perfomance impact of NumPy API change

[Matti Picus]

Breaking this into a number of sub-dsicussions, since we seem to be 
branching. The original topic was


Re: [Cython] Enhancing "ctyepdef class numpy.ndarray" with getter properties

On 28/09/18 01:20, Robert Bradshaw wrote:


Hmm...so in this case it upgrading Cython would cause an unconditional 
switch from direct access to a function call without any code change 
(or choice) for users of numpy.pxd. I am curious what kind of a 
slowdown this would represent (though would assume this kind of 
analysis was done by the NumPy folks when choosing macro vs. function 
for the public API).


As I point out in the "experiment" comment referenced above,
pandas has
code that needs lvalue access to ndarray data, so they would be stuck
with the old API which is deprecated but still works for now.
Scipy has
no such code and oculd move forward to the newer API.


But if we upgraded Cython, how would they access the old API? I 
suppose they could create a setter macro of their own to use in the 
(presumably few) cases where they needed an lvalue.


- Robert




NumPy changed its recommended API to an opaque one via inline getter 
functions in 2011, in this PR https://github.com/numpy/numpy/pull/116. I 
could not find a discussion on performance impact, perhaps since the 
functions are in the header files and marked inline. Hopefully the 
compilers will properly deal with making them fast. However, it is true 
that when people update to a new version of a library things change. In 
this case, there are backward-compatibility macros that revert the 
post-1.7 functions into pre-1.7 macros with the same name.


Thus for the experiment I used a new numpy.pxd, defined the pre-1.7 api 
in the pandas build (experimental changeset 
https://github.com/mattip/pandas/commit/9113bf7e55e1eddece3544c1ad3ef2a761b5210a), 
and was still able to access ndarray.data as a lvalue.


Matti
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Enhancing "ctyepdef class numpy.ndarray" with getter properties

[Matti Picus]

On 28/09/18 01:20, Robert Bradshaw wrote:
On Thu, Sep 27, 2018 at 11:36 PM Matti Picus > wrote:


The problem is that when one reads

    cdef int aaa 

there's no indication as to the meaning of this. We also want to be 
sure to disallow this syntax everywhere but this one context. On the 
other hand the quotation syntax


    cdef int aaa "bbb"

already has (widespread) meaning of establishing a C alias of the name 
in question which is essentially what we're trying to do here.


I'm still, however, leaning towards the @property syntax (which we 
could allow for non-extern cdef classes as well).


- Robert


Using "PyArray_DIMS" with quotes but without parentheses would indeed be 
confusing to users and difficult to implement, so "PyArray_DIMS(*)" 
where the * is TBD seem nicer.


It sounds like the jury is still out. In order to compare the solutions, 
I will move forward with the @property decorator syntax, but to keep it 
simple I will start small: only getters and specifically for 
CFuncDefNodes. Then if you still want to look at the other option I will 
turn my "experiment into a PR.


Matti
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Mitigating perfomance impact of NumPy API change

[Matti Picus]

On 28/09/18 10:25, Matti Picus wrote:
Breaking this into a number of sub-dsicussions, since we seem to be 
branching. The original topic was


Re: [Cython] Enhancing "ctyepdef class numpy.ndarray" with getter 
properties


On 28/09/18 01:20, Robert Bradshaw wrote:


Hmm...so in this case it upgrading Cython would cause an 
unconditional switch from direct access to a function call without 
any code change (or choice) for users of numpy.pxd. I am curious what 
kind of a slowdown this would represent (though would assume this 
kind of analysis was done by the NumPy folks when choosing macro vs. 
function for the public API).


    As I point out in the "experiment" comment referenced above,
    pandas has
    code that needs lvalue access to ndarray data, so they would be 
stuck

    with the old API which is deprecated but still works for now.
    Scipy has
    no such code and oculd move forward to the newer API.


But if we upgraded Cython, how would they access the old API? I 
suppose they could create a setter macro of their own to use in the 
(presumably few) cases where they needed an lvalue.


- Robert




NumPy changed its recommended API to an opaque one via inline getter 
functions in 2011, in this PR https://github.com/numpy/numpy/pull/116. 
I could not find a discussion on performance impact, perhaps since the 
functions are in the header files and marked inline. Hopefully the 
compilers will properly deal with making them fast. However, it is 
true that when people update to a new version of a library things 
change. In this case, there are backward-compatibility macros that 
revert the post-1.7 functions into pre-1.7 macros with the same name.


Thus for the experiment I used a new numpy.pxd, defined the pre-1.7 
api in the pandas build (experimental changeset 
https://github.com/mattip/pandas/commit/9113bf7e55e1eddece3544c1ad3ef2a761b5210a), 
and was still able to access ndarray.data as a lvalue.


Matti


This means cython/numpy could provide an integration path based on numpy 
starting to ship its own numpy.pxd:


- Cython would define the macro (if not already defined) to use the 
pre-1.7 Numpy API in the numpy.pxd it ships. This would still work 
(lvalues would be allowed) after direct access is replaced with the 
getter properties, since they are macros


- NumPy would define the macro to use post-1.7 API (if not already 
defined) in the numpy.pxd it ships, which as I understand would take 
precedence over cython's. Then projects like pandas could freely upgrade 
Cython without changing their codebase, but would encounter errors when 
updating NumPy.


Matti
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Preparing the language level change

[Stefan Behnel]

Stefan Behnel schrieb am 21.09.2018 um 09:38:
> There are two parts of
> information here, so maybe we should actually split them internally (in
> "Main.Context.set_language_level() ?) and keep the language_level = 3 but
> just avoid the "unicode_literals" part.

I thought about this some more. The real question is: what should the world
be like once Cython 3.0 is out?

What we want to do for Cy3 is to change the default behaviour, which mostly
impacts Py2/3 ported code. That's why we now warn about a missing
"language_level" switch. When people respond to that and set the language
level explicitly, that makes them opt out of the default change. Perfect so
far.

Do we then still want to have a separate option floating around that says
"but I want str" ? I don't think so, because that will be the default in
Cy3 anyway. Thus, what I think we want is that people either specify the
language level explicitly, or get the new default. And to get the new
default *now*, in a future proof way, I think the best option is to set an
explicit language level, not a separate directive.

Thus, I now agree with Jeroen's early intuition that a new language level
switch is the right interface. I'll change the implementation to do what I
wrote in the quoted paragraph above and push a new RC.

Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel