[Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
[moving to dev list] On 05/07/2012 11:17 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 10:44: On 05/07/2012 07:48 AM, Stefan Behnel wrote: shaunc, 07.05.2012 07:13: The following code: cdef int foo( double[:] bar ) nogil: return bar is None causes: "Converting to Python object not allowed without gil" However, I was under the impression that: "When comparing a value with None, keep in mind that, if x is a Python object, x is None and x is not None are very efficient because they translate directly to C pointer comparisons," I guess the problem is that the memoryview is not a python object -- indeed, this compiles in the form: cdef int foo( object bar ) nogil: return bar is None But this is a bit counterintuitive... do I need to do "with gil" to check if a memoryview is None? And in a nogil function, I'm not necessarily guaranteed that I don't have the gil -- what is the best way ensure I have the gil? (Is there a "secret system call" or should I use a try block?) It would seem more appropriate (IMHO, of course :)) to allow "bar is None" also when bar is a memoryview I wonder why a memory view should be allowed to be None in the first place. Buffer arguments aren't (because they get unpacked on entry), so why should memory views? ? At least when I implemented it, buffers get unpacked but the case of a None buffer is treated specially, and you're fully allowed (and segfault if you [] it). Hmm, ok, maybe I just got confused by the code then. I think the docs should state that buffer arguments are best used together with the "not None" declaration then. I use them with "=None" default values all the time... then do a None-check manually. It's really no different from cdef classes. And I remember that we wanted to change the default settings for extension type arguments from "or None" to "not None" years ago but never actually did it. I remember that there was such a debate, but I certainly don't remember that this was the conclusion :-) I didn't agree with that view then and I don't now. I don't remember what Robert's view was... As far as I can remember (which might be biased towards my personal view), the conclusion was that we left the current semantics in place, relying on better control flow analysis to make None-checks cheaper, and when those are cheap enough, make the nonecheck directive default to True (Java is sort of prior art that this can indeed be done?). Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
Dag Sverre Seljebotn, 07.05.2012 12:40: > moving to dev list Makes sense. > On 05/07/2012 11:17 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 07.05.2012 10:44: >>> On 05/07/2012 07:48 AM, Stefan Behnel wrote: I wonder why a memory view should be allowed to be None in the first place. Buffer arguments aren't (because they get unpacked on entry), so why should memory views? >>> >>> ? At least when I implemented it, buffers get unpacked but the case of a >>> None buffer is treated specially, and you're fully allowed (and segfault if >>> you [] it). >> >> Hmm, ok, maybe I just got confused by the code then. >> >> I think the docs should state that buffer arguments are best used together >> with the "not None" declaration then. ... which made me realise that that wasn't even supported. I can't believe no-one ever reported that as a bug... https://github.com/cython/cython/commit/f2de49fd0ac82a02a070b931bf4d2dab47135d0b It's still not supported for memory views. BTW, is there a reason why we shouldn't allow a "not None" declaration for cdef functions? Obviously, the caller would have to do the check in that case. Hmm, maybe it's not that important, because None checks are best done at entry points from user code, which usually means Python code. It seems like "not None" is not supported on cpdef functions, though. > I use them with "=None" default values all the time... then do a > None-check manually. Interesting. Could you given an example? What's the advantage over letting Cython raise an error for you? And, since you are using it as a default argument, why would someone want to call your code entirely without a buffer argument? > It's really no different from cdef classes. I find it at least a bit more surprising because a buffer unpacking argument is a rather strong hint that you expect something that supports this protocol. The fact that you type your function argument with it hints at the intention to properly unpack it on entry. I'm sure there are lots of users who were or will be surprised when they realise that that doesn't exclude None values. >> And I remember that we wanted to change the default settings for extension >> type arguments from "or None" to "not None" years ago but never actually >> did it. > > I remember that there was such a debate, but I certainly don't remember > that this was the conclusion :-) Maybe not, yes. > I didn't agree with that view then and > I don't now. I don't remember what Robert's view was... > > As far as I can remember (which might be biased towards my personal > view), the conclusion was that we left the current semantics in place, > relying on better control flow analysis to make None-checks cheaper, and > when those are cheap enough, make the nonecheck directive default to > True At least for buffer arguments, it silently corrupts data or segfaults in the current state of affairs, as you pointed out. Not exactly ideal. That's another reason why I see a difference between the behaviour of extension types and that of buffer arguments. Buffer indexing is also way more performance critical than the average method call or attribute access on a cdef class. > (Java is sort of prior art that this can indeed be done?). Java was designed to have a JIT compiler underneath which handles external parameters, and its compilers are way smarter than Cython. I agree that there is still a lot we can do based on better static analysis, but there will always be limits. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 05/07/2012 01:10 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 12:40: moving to dev list Makes sense. On 05/07/2012 11:17 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 10:44: On 05/07/2012 07:48 AM, Stefan Behnel wrote: I wonder why a memory view should be allowed to be None in the first place. Buffer arguments aren't (because they get unpacked on entry), so why should memory views? ? At least when I implemented it, buffers get unpacked but the case of a None buffer is treated specially, and you're fully allowed (and segfault if you [] it). Hmm, ok, maybe I just got confused by the code then. I think the docs should state that buffer arguments are best used together with the "not None" declaration then. ... which made me realise that that wasn't even supported. I can't believe no-one ever reported that as a bug... https://github.com/cython/cython/commit/f2de49fd0ac82a02a070b931bf4d2dab47135d0b It's still not supported for memory views. BTW, is there a reason why we shouldn't allow a "not None" declaration for cdef functions? Obviously, the caller would have to do the check in that case. Hmm, maybe it's not that important, because None checks are best done at entry points from user code, which usually means Python code. It seems like "not None" is not supported on cpdef functions, though. I use them with "=None" default values all the time... then do a None-check manually. Interesting. Could you given an example? What's the advantage over letting Cython raise an error for you? And, since you are using it as a default argument, why would someone want to call your code entirely without a buffer argument? Here you go: def foo(np.ndarray[double] a, np.ndarray[double] out=None): if out is None: out = np.empty_like(a) # compute result in out return out The pattern of handing in the memory area to write to is one of the fundamental basics of numerical computing; you often just can't implement an algorithm if the called function returns the result in a newly-allocated array. I can explain why that is in detail, but I'd rather you just trusted the testimony of somebody doing numerical computation... It's just a convenience, but often (in particular when testing) it's incredibly convenient to not have to bother with allocating the output array. Another pattern is: def do_something(np.ndarray[double] a, np.ndarray[double] sin_of_a=None): ... so if your caller happened to already have computed something, the function uses it, but OTOH the "something" is a function of the inputs and can be computed on the fly. AND, sometimes it can be computed on the fly in ways more efficient than what the caller could have done, because of memory bus issues etc. etc. Both of these can be "fixed" by a) not allowing the convenient shorthand, or b) declare the argument "object" first and then type it after the "preamble". So the REAL reason I'm arguing this case is consistency with cdef classes. It's really no different from cdef classes. I find it at least a bit more surprising because a buffer unpacking argument is a rather strong hint that you expect something that supports this protocol. The fact that you type your function argument with it hints at the intention to properly unpack it on entry. I'm sure there are lots of users who were or will be surprised when they realise that that doesn't exclude None values. Whereas I think there would be more users surprised by the opposite. So there -- we won't know who's right without actually finding some users. And chances are we are both right, since users are different from one another. And I remember that we wanted to change the default settings for extension type arguments from "or None" to "not None" years ago but never actually did it. I remember that there was such a debate, but I certainly don't remember that this was the conclusion :-) Maybe not, yes. I didn't agree with that view then and I don't now. I don't remember what Robert's view was... As far as I can remember (which might be biased towards my personal view), the conclusion was that we left the current semantics in place, relying on better control flow analysis to make None-checks cheaper, and when those are cheap enough, make the nonecheck directive default to True At least for buffer arguments, it silently corrupts data or segfaults in the current state of affairs, as you pointed out. Not exactly ideal. No different than writing to a field in a cdef class... That's another reason why I see a difference between the behaviour of extension types and that of buffer arguments. Buffer indexing is also way more performance critical than the average method call or attribute access on a cdef class. Perhaps, but that's a bit hand-wavy to turn into a principle of language design? "This is performance critical, so therefore we suddenly invert the normal rule"? I just think we
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 05/07/2012 01:48 PM, Dag Sverre Seljebotn wrote: On 05/07/2012 01:10 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 12:40: moving to dev list Makes sense. On 05/07/2012 11:17 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 10:44: On 05/07/2012 07:48 AM, Stefan Behnel wrote: I wonder why a memory view should be allowed to be None in the first place. Buffer arguments aren't (because they get unpacked on entry), so why should memory views? ? At least when I implemented it, buffers get unpacked but the case of a None buffer is treated specially, and you're fully allowed (and segfault if you [] it). Hmm, ok, maybe I just got confused by the code then. I think the docs should state that buffer arguments are best used together with the "not None" declaration then. ... which made me realise that that wasn't even supported. I can't believe no-one ever reported that as a bug... https://github.com/cython/cython/commit/f2de49fd0ac82a02a070b931bf4d2dab47135d0b It's still not supported for memory views. BTW, is there a reason why we shouldn't allow a "not None" declaration for cdef functions? Obviously, the caller would have to do the check in that case. Hmm, maybe it's not that important, because None checks are best done at entry points from user code, which usually means Python code. It seems like "not None" is not supported on cpdef functions, though. I use them with "=None" default values all the time... then do a None-check manually. Interesting. Could you given an example? What's the advantage over letting Cython raise an error for you? And, since you are using it as a default argument, why would someone want to call your code entirely without a buffer argument? Here you go: def foo(np.ndarray[double] a, np.ndarray[double] out=None): if out is None: out = np.empty_like(a) # compute result in out return out The pattern of handing in the memory area to write to is one of the fundamental basics of numerical computing; you often just can't implement an algorithm if the called function returns the result in a newly-allocated array. I can explain why that is in detail, but I'd rather you just trusted the testimony of somebody doing numerical computation... It's just a convenience, but often (in particular when testing) it's incredibly convenient to not have to bother with allocating the output array. Another pattern is: def do_something(np.ndarray[double] a, np.ndarray[double] sin_of_a=None): ... so if your caller happened to already have computed something, the function uses it, but OTOH the "something" is a function of the inputs and can be computed on the fly. AND, sometimes it can be computed on the fly in ways more efficient than what the caller could have done, because of memory bus issues etc. etc. Both of these can be "fixed" by a) not allowing the convenient shorthand, or b) declare the argument "object" first and then type it after the "preamble". So the REAL reason I'm arguing this case is consistency with cdef classes. It's really no different from cdef classes. I find it at least a bit more surprising because a buffer unpacking argument is a rather strong hint that you expect something that supports this protocol. The fact that you type your function argument with it hints at the intention to properly unpack it on entry. I'm sure there are lots of users who were or will be surprised when they realise that that doesn't exclude None values. Whereas I think there would be more users surprised by the opposite. So there -- we won't know who's right without actually finding some users. And chances are we are both right, since users are different from one another. And I remember that we wanted to change the default settings for extension type arguments from "or None" to "not None" years ago but never actually did it. I remember that there was such a debate, but I certainly don't remember that this was the conclusion :-) Maybe not, yes. I didn't agree with that view then and I don't now. I don't remember what Robert's view was... As far as I can remember (which might be biased towards my personal view), the conclusion was that we left the current semantics in place, relying on better control flow analysis to make None-checks cheaper, and when those are cheap enough, make the nonecheck directive default to True At least for buffer arguments, it silently corrupts data or segfaults in the current state of affairs, as you pointed out. Not exactly ideal. No different than writing to a field in a cdef class... Also, I believe that in the strided case, the strides are all set to 0, and the data-pointer is NULL, so you will never corrupt data, you will always try to access *NULL and segfault. Though If you put mode='c' and a very high index you'll corrupt data. Dag That's another reason why I see a difference between the behaviour of extension types and that of buffer arguments. Buffer indexing is also way more performanc
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
Dag Sverre Seljebotn, 07.05.2012 13:48: > On 05/07/2012 01:10 PM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 07.05.2012 12:40: >>> On 05/07/2012 11:17 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 10:44: > On 05/07/2012 07:48 AM, Stefan Behnel wrote: >> I wonder why a memory view should be allowed to be None in the first >> place. >> Buffer arguments aren't (because they get unpacked on entry), so why >> should memory views? > > ? At least when I implemented it, buffers get unpacked but the case of a > None buffer is treated specially, and you're fully allowed (and > segfault if you [] it). Hmm, ok, maybe I just got confused by the code then. I think the docs should state that buffer arguments are best used together with the "not None" declaration then. >>> >>> I use them with "=None" default values all the time... then do a >>> None-check manually. >> >> Interesting. Could you given an example? What's the advantage over letting >> Cython raise an error for you? And, since you are using it as a default >> argument, why would someone want to call your code entirely without a >> buffer argument? > > Here you go: > > def foo(np.ndarray[double] a, np.ndarray[double] out=None): > if out is None: > out = np.empty_like(a) Ah, right - output arguments. Hadn't thought of those. Still, since you pass None explicitly as a default argument, this code wouldn't be impacted by disallowing None for buffers by default. That case is already handled specially in the compiler. But a better default would prevent the *first* argument from being None. So, basically, it would do the right thing straight away in your case and generate safer and more efficient code for it, whereas now you have to test 'a' for being None explicitly and Cython won't understand that hint due to insufficient static analysis. At least, since my last commit you can make Cython do the same thing by declaring it "not None". >>> It's really no different from cdef classes. >> >> I find it at least a bit more surprising because a buffer unpacking >> argument is a rather strong hint that you expect something that supports >> this protocol. The fact that you type your function argument with it hints >> at the intention to properly unpack it on entry. I'm sure there are lots of >> users who were or will be surprised when they realise that that doesn't >> exclude None values. > > Whereas I think there would be more users surprised by the opposite. We've had enough complaints from users about None being allowed for typed arguments already to consider it at least a gotcha of the language. The main reason we didn't change this behaviour back then was that it would clearly break user code and we thought we could do without that. That's different from considering it "right" and "good". And I remember that we wanted to change the default settings for extension type arguments from "or None" to "not None" years ago but never actually did it. >>> >>> I remember that there was such a debate, but I certainly don't remember >>> that this was the conclusion :-) >> >> Maybe not, yes. >> >> >>> I didn't agree with that view then and >>> I don't now. I don't remember what Robert's view was... >>> >>> As far as I can remember (which might be biased towards my personal >>> view), the conclusion was that we left the current semantics in place, >>> relying on better control flow analysis to make None-checks cheaper, and >>> when those are cheap enough, make the nonecheck directive default to >>> True >> >> At least for buffer arguments, it silently corrupts data or segfaults in >> the current state of affairs, as you pointed out. Not exactly ideal. > > No different than writing to a field in a cdef class... Hmm, aren't those None checked? At least cdef method calls are AFAIR. I think we should really get back to the habit of making code safe first and fast afterwards. >> That's another reason why I see a difference between the behaviour of >> extension types and that of buffer arguments. Buffer indexing is also way >> more performance critical than the average method call or attribute access >> on a cdef class. > > Perhaps, but that's a bit hand-wavy to turn into a principle of language > design? "This is performance critical, so therefore we suddenly invert the > normal rule"? Since when is the "normal rule" to consider performance so important that we prefer a crash over raising an exception? That's the current state of buffer arguments, after all, so we already inverted the "normal rule", IMHO. > I just think we should be consistent, not have more special rules for > buffers than we need to. Agreed. So, would you accept that we add a None check to every buffer indexing access now and try to eliminate them over time (or with user interaction)? > The intention all the time was that "np.ndarray[double]" is just a > glorified "np.ndarray". People expect
[Cython] buffer syntax vs. memory view syntax (was: Re: checking for "None" in nogil function)
Stefan Behnel, 07.05.2012 15:04: > Dag Sverre Seljebotn, 07.05.2012 13:48: >> BTW, with the coming of memoryviews, me and Mark talked about just >> deprecating the "mytype[...]" meaning buffers, and rather treat it as >> np.ndarray, array.array etc. being some sort of "template types". That is, >> we disallow "object[int]" and require some special declarations in the >> relevant pxd files. > > Hmm, yes, it's unfortunate that we have two different types of syntax now, > one that declares the item type before the brackets and one that declares > it afterwards. I actually think this merits some more discussion. Should we consider the buffer interface syntax deprecated and focus on the memory view syntax? The words-to-punctuation ratio of the latter may hurt the eyes when encountering it unprepared, but at least it doesn't require two type names, of which the one before the brackets (i.e. "object") is mostly useless. (Although it does reflect the notion that we are dealing with an object here ...) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] How do you trigger a jenkins build?
I've noticed that old one URL hook doesn't work for me now. I tried to check "Build when a change is pushed to GitHub" and set "Jenkins Hook URL" to https://sage.math.washington.edu:8091/hudson/github-webhook/ That doesn't work. What is the right way? -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 05/07/2012 03:04 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 13:48: Here you go: def foo(np.ndarray[double] a, np.ndarray[double] out=None): if out is None: out = np.empty_like(a) Ah, right - output arguments. Hadn't thought of those. Still, since you pass None explicitly as a default argument, this code wouldn't be impacted by disallowing None for buffers by default. That case is already handled specially in the compiler. But a better default would prevent the *first* argument from being None. So, basically, it would do the right thing straight away in your case and generate safer and more efficient code for it, whereas now you have to test 'a' for being None explicitly and Cython won't understand that hint due to insufficient static analysis. At least, since my last commit you can make Cython do the same thing by declaring it "not None". Yes, thanks! It's really no different from cdef classes. I find it at least a bit more surprising because a buffer unpacking argument is a rather strong hint that you expect something that supports this protocol. The fact that you type your function argument with it hints at the intention to properly unpack it on entry. I'm sure there are lots of users who were or will be surprised when they realise that that doesn't exclude None values. Whereas I think there would be more users surprised by the opposite. We've had enough complaints from users about None being allowed for typed arguments already to consider it at least a gotcha of the language. The main reason we didn't change this behaviour back then was that it would clearly break user code and we thought we could do without that. That's different from considering it "right" and "good". And I remember that we wanted to change the default settings for extension type arguments from "or None" to "not None" years ago but never actually did it. I remember that there was such a debate, but I certainly don't remember that this was the conclusion :-) Maybe not, yes. I didn't agree with that view then and I don't now. I don't remember what Robert's view was... As far as I can remember (which might be biased towards my personal view), the conclusion was that we left the current semantics in place, relying on better control flow analysis to make None-checks cheaper, and when those are cheap enough, make the nonecheck directive default to True At least for buffer arguments, it silently corrupts data or segfaults in the current state of affairs, as you pointed out. Not exactly ideal. No different than writing to a field in a cdef class... Hmm, aren't those None checked? At least cdef method calls are AFAIR. Not at all. That's my whole point -- currently, the rule for None in Cython is "it's your responsibility to never do a native operation on None". I don't like that either, but that's just inherited from Pyrex (and many projects would get speed regressions etc.). I'm not against changing that to "we safely None-check", if done nicely -- it's just that that should be done everywhere at once. In current master (and as far back as I can remember), this code: cdef class A: cdef int field cdef int method(self): print self.field def f(): cdef A a = None a.field = 3 a.method() Turns into: __pyx_v_a = ((struct __pyx_obj_5test2_A *)Py_None); __pyx_v_a->field = 3; ((struct __pyx_vtabstruct_5test2_A *) __pyx_v_a->__pyx_vtab)->method(__pyx_v_a); I think we should really get back to the habit of making code safe first and fast afterwards. Nobody has argued otherwise for some time (since the cdivision thread I believe), this is all about Pyrex legacy. Guess part of the story is that there's lots of performance-sensitive code in SAGE using cdef classes which was written in Pyrex before Cython was around... In fact, the nonecheck directive was written by yours truly! And I argued for making it the default at the time! Because it uses syntax that is expected to unpack the buffer. If that buffer doesn't exist, I'd expect an error. It's like using interfaces: I want something here that implements the buffer interface. If it doesn't - reject it. Besides, I hope you are aware that your argumentation stands on the (IMHO questionable) fact that "np.ndarray" by itself can be None by default. If np.ndarray should not be be allowed to be None by default, why should np.ndarray[double]? That argument works in both ways. I'm well aware of it... Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
On 05/07/2012 04:16 PM, Stefan Behnel wrote: Stefan Behnel, 07.05.2012 15:04: Dag Sverre Seljebotn, 07.05.2012 13:48: BTW, with the coming of memoryviews, me and Mark talked about just deprecating the "mytype[...]" meaning buffers, and rather treat it as np.ndarray, array.array etc. being some sort of "template types". That is, we disallow "object[int]" and require some special declarations in the relevant pxd files. Hmm, yes, it's unfortunate that we have two different types of syntax now, one that declares the item type before the brackets and one that declares it afterwards. I actually think this merits some more discussion. Should we consider the buffer interface syntax deprecated and focus on the memory view syntax? I think that's the very-long-term intention. Then again, it may be too early to really tell yet, we just need to see how the memory views play out in real life and whether they'll be able to replace np.ndarray[double] among real users. We don't want to shove things down users throats. But the use of the trailing-[] syntax needs some cleaning up. Me and Mark agreed we'd put this proposal forward when we got around to it: - Deprecate the "object[double]" form, where [dtype] can be stuck on any extension type - But, do NOT (for the next year at least) deprecate np.ndarray[double], array.array[double], etc. Basically, there should be a magic flag in extension type declarations saying "I can be a buffer". For one thing, that is sort of needed to open up things for templated cdef classes/fused types cdef classes, if that is ever implemented. The semantic meaning of trailing [] is still sort of like the C++ meaning; that it templates the argument types (except it's lots of special cases in the compiler for various things rather than a Turing-complete template language...) Dag The words-to-punctuation ratio of the latter may hurt the eyes when encountering it unprepared, but at least it doesn't require two type names, of which the one before the brackets (i.e. "object") is mostly useless. (Although it does reflect the notion that we are dealing with an object here ...) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
On 05/07/2012 06:00 PM, Dag Sverre Seljebotn wrote: On 05/07/2012 04:16 PM, Stefan Behnel wrote: Stefan Behnel, 07.05.2012 15:04: Dag Sverre Seljebotn, 07.05.2012 13:48: BTW, with the coming of memoryviews, me and Mark talked about just deprecating the "mytype[...]" meaning buffers, and rather treat it as np.ndarray, array.array etc. being some sort of "template types". That is, we disallow "object[int]" and require some special declarations in the relevant pxd files. Hmm, yes, it's unfortunate that we have two different types of syntax now, one that declares the item type before the brackets and one that declares it afterwards. I actually think this merits some more discussion. Should we consider the buffer interface syntax deprecated and focus on the memory view syntax? I think that's the very-long-term intention. Then again, it may be too early to really tell yet, we just need to see how the memory views play out in real life and whether they'll be able to replace np.ndarray[double] among real users. We don't want to shove things down users throats. But the use of the trailing-[] syntax needs some cleaning up. Me and Mark agreed we'd put this proposal forward when we got around to it: - Deprecate the "object[double]" form, where [dtype] can be stuck on any extension type - But, do NOT (for the next year at least) deprecate np.ndarray[double], array.array[double], etc. Basically, there should be a magic flag in extension type declarations saying "I can be a buffer". For one thing, that is sort of needed to open up things for templated cdef classes/fused types cdef classes, if that is ever implemented. The semantic meaning of trailing [] is still sort of like the C++ meaning; that it templates the argument types (except it's lots of special cases in the compiler for various things rather than a Turing-complete template language...) s/argument types/base type/ Dag Dag The words-to-punctuation ratio of the latter may hurt the eyes when encountering it unprepared, but at least it doesn't require two type names, of which the one before the brackets (i.e. "object") is mostly useless. (Although it does reflect the notion that we are dealing with an object here ...) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 7 May 2012 12:51, Dag Sverre Seljebotn wrote: > On 05/07/2012 01:48 PM, Dag Sverre Seljebotn wrote: >> >> On 05/07/2012 01:10 PM, Stefan Behnel wrote: >>> >>> Dag Sverre Seljebotn, 07.05.2012 12:40: moving to dev list >>> >>> >>> Makes sense. >>> On 05/07/2012 11:17 AM, Stefan Behnel wrote: > > Dag Sverre Seljebotn, 07.05.2012 10:44: >> >> On 05/07/2012 07:48 AM, Stefan Behnel wrote: >>> >>> I wonder why a memory view should be allowed to be None in the first >>> place. >>> Buffer arguments aren't (because they get unpacked on entry), so why >>> should memory views? >> >> >> ? At least when I implemented it, buffers get unpacked but the case >> of a >> None buffer is treated specially, and you're fully allowed (and >> segfault if >> you [] it). > > > Hmm, ok, maybe I just got confused by the code then. > > I think the docs should state that buffer arguments are best used > together > with the "not None" declaration then. >>> >>> >>> ... which made me realise that that wasn't even supported. I can't >>> believe >>> no-one ever reported that as a bug... >>> >>> >>> https://github.com/cython/cython/commit/f2de49fd0ac82a02a070b931bf4d2dab47135d0b >>> >>> >>> It's still not supported for memory views. >>> >>> BTW, is there a reason why we shouldn't allow a "not None" declaration >>> for >>> cdef functions? Obviously, the caller would have to do the check in that >>> case. Hmm, maybe it's not that important, because None checks are best >>> done >>> at entry points from user code, which usually means Python code. It seems >>> like "not None" is not supported on cpdef functions, though. >>> >>> I use them with "=None" default values all the time... then do a None-check manually. >>> >>> >>> Interesting. Could you given an example? What's the advantage over >>> letting >>> Cython raise an error for you? And, since you are using it as a default >>> argument, why would someone want to call your code entirely without a >>> buffer argument? >> >> >> Here you go: >> >> def foo(np.ndarray[double] a, np.ndarray[double] out=None): >> if out is None: >> out = np.empty_like(a) >> # compute result in out >> return out >> >> The pattern of handing in the memory area to write to is one of the >> fundamental basics of numerical computing; you often just can't >> implement an algorithm if the called function returns the result in a >> newly-allocated array. I can explain why that is in detail, but I'd >> rather you just trusted the testimony of somebody doing numerical >> computation... >> >> It's just a convenience, but often (in particular when testing) it's >> incredibly convenient to not have to bother with allocating the output >> array. >> >> Another pattern is: >> >> def do_something(np.ndarray[double] a, >> np.ndarray[double] sin_of_a=None): >> ... >> >> so if your caller happened to already have computed something, the >> function uses it, but OTOH the "something" is a function of the inputs >> and can be computed on the fly. AND, sometimes it can be computed on the >> fly in ways more efficient than what the caller could have done, because >> of memory bus issues etc. etc. >> >> Both of these can be "fixed" by a) not allowing the convenient >> shorthand, or b) declare the argument "object" first and then type it >> after the "preamble". >> >> So the REAL reason I'm arguing this case is consistency with cdef classes. >> >> >> >>> >>> It's really no different from cdef classes. >>> >>> >>> I find it at least a bit more surprising because a buffer unpacking >>> argument is a rather strong hint that you expect something that supports >>> this protocol. The fact that you type your function argument with it >>> hints >>> at the intention to properly unpack it on entry. I'm sure there are >>> lots of >>> users who were or will be surprised when they realise that that doesn't >>> exclude None values. >> >> >> Whereas I think there would be more users surprised by the opposite. >> >> So there -- we won't know who's right without actually finding some >> users. And chances are we are both right, since users are different from >> one another. >> >>> >>> > And I remember that we wanted to change the default settings for > extension > type arguments from "or None" to "not None" years ago but never > actually > did it. I remember that there was such a debate, but I certainly don't remember that this was the conclusion :-) >>> >>> >>> Maybe not, yes. >>> >>> I didn't agree with that view then and I don't now. I don't remember what Robert's view was... As far as I can remember (which might be biased towards my personal view), the conclusion was that we left the current semantics in place, relying on better control flow analysis to make None-checks cheaper, and when those are cheap enough, make the nonecheck directive default to True
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 7 May 2012 12:10, Stefan Behnel wrote: > Dag Sverre Seljebotn, 07.05.2012 12:40: >> moving to dev list > > Makes sense. > >> On 05/07/2012 11:17 AM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn, 07.05.2012 10:44: On 05/07/2012 07:48 AM, Stefan Behnel wrote: > I wonder why a memory view should be allowed to be None in the first > place. > Buffer arguments aren't (because they get unpacked on entry), so why > should memory views? ? At least when I implemented it, buffers get unpacked but the case of a None buffer is treated specially, and you're fully allowed (and segfault if you [] it). >>> >>> Hmm, ok, maybe I just got confused by the code then. >>> >>> I think the docs should state that buffer arguments are best used together >>> with the "not None" declaration then. > > ... which made me realise that that wasn't even supported. I can't believe > no-one ever reported that as a bug... > > https://github.com/cython/cython/commit/f2de49fd0ac82a02a070b931bf4d2dab47135d0b > > It's still not supported for memory views. Yeah, that was never implemented, but probably should be. > BTW, is there a reason why we shouldn't allow a "not None" declaration for > cdef functions? Obviously, the caller would have to do the check in that > case. Why can't the callee just check it? If it's None, just raise an exception like usual? > Hmm, maybe it's not that important, because None checks are best done > at entry points from user code, which usually means Python code. It seems > like "not None" is not supported on cpdef functions, though. > > >> I use them with "=None" default values all the time... then do a >> None-check manually. > > Interesting. Could you given an example? What's the advantage over letting > Cython raise an error for you? And, since you are using it as a default > argument, why would someone want to call your code entirely without a > buffer argument? > > >> It's really no different from cdef classes. > > I find it at least a bit more surprising because a buffer unpacking > argument is a rather strong hint that you expect something that supports > this protocol. The fact that you type your function argument with it hints > at the intention to properly unpack it on entry. I'm sure there are lots of > users who were or will be surprised when they realise that that doesn't > exclude None values. > > >>> And I remember that we wanted to change the default settings for extension >>> type arguments from "or None" to "not None" years ago but never actually >>> did it. >> >> I remember that there was such a debate, but I certainly don't remember >> that this was the conclusion :-) > > Maybe not, yes. > > >> I didn't agree with that view then and >> I don't now. I don't remember what Robert's view was... >> >> As far as I can remember (which might be biased towards my personal >> view), the conclusion was that we left the current semantics in place, >> relying on better control flow analysis to make None-checks cheaper, and >> when those are cheap enough, make the nonecheck directive default to >> True > > At least for buffer arguments, it silently corrupts data or segfaults in > the current state of affairs, as you pointed out. Not exactly ideal. > > That's another reason why I see a difference between the behaviour of > extension types and that of buffer arguments. Buffer indexing is also way > more performance critical than the average method call or attribute access > on a cdef class. > > >> (Java is sort of prior art that this can indeed be done?). > > Java was designed to have a JIT compiler underneath which handles external > parameters, and its compilers are way smarter than Cython. I agree that > there is still a lot we can do based on better static analysis, but there > will always be limits. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 0.17
2012/5/7 mark florisson : > On 6 May 2012 20:41, Matthew Brett wrote: >> Hi, >> >> On Sun, May 6, 2012 at 7:28 AM, mark florisson >> wrote: >>> Hey, >>> >>> I think we already have quite a bit of functionality (nearly) ready, >>> after merging some pending pull requests maybe it will be a good time >>> for a 0.17 release? I think it would be good to also document to what >>> extent pypy support works, what works and what doesn't. Stefan, since >>> you added a large majority of the features, would you want to be the >>> release manager? >>> >>> In summary, the following pull requests should likely go in >>> - array.array support (unless further discussion prevents that) >>> - fused types runtime buffer dispatch >>> - newaxis >>> - more? >>> >>> The memoryview documentation should also be reworked a bit. Matthew, >>> are you still willing to have a go at that? Otherwise I can clean up >>> the mess first, some things are no longer true and simply outdated, >>> and then have a second opinion. >> >> Yes, sorry, I have been taken up by releasing my own project. What's >> the deadline do you think? I have another big release to do for the >> end of next week, but I might be able to carve out some time, >> >> See you, >> >> Matthew > > Great, I'd say we're probably not going to release anything within the > next two weeks, so take your time, there is no hurry really :). Hmm, it seems to me that master is currently broken: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py27-ext/ -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 05/07/2012 06:04 PM, mark florisson wrote: On 7 May 2012 12:10, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 12:40: moving to dev list Makes sense. On 05/07/2012 11:17 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 10:44: On 05/07/2012 07:48 AM, Stefan Behnel wrote: I wonder why a memory view should be allowed to be None in the first place. Buffer arguments aren't (because they get unpacked on entry), so why should memory views? ? At least when I implemented it, buffers get unpacked but the case of a None buffer is treated specially, and you're fully allowed (and segfault if you [] it). Hmm, ok, maybe I just got confused by the code then. I think the docs should state that buffer arguments are best used together with the "not None" declaration then. ... which made me realise that that wasn't even supported. I can't believe no-one ever reported that as a bug... https://github.com/cython/cython/commit/f2de49fd0ac82a02a070b931bf4d2dab47135d0b It's still not supported for memory views. Yeah, that was never implemented, but probably should be. BTW, is there a reason why we shouldn't allow a "not None" declaration for cdef functions? Obviously, the caller would have to do the check in that case. Why can't the callee just check it? If it's None, just raise an exception like usual? It's just that there's a lot more potential for rather easy optimization if the caller does it. Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
Dag Sverre Seljebotn, 07.05.2012 18:07: > On 05/07/2012 06:04 PM, mark florisson wrote: >> On 7 May 2012 12:10, Stefan Behnel wrote: >>> BTW, is there a reason why we shouldn't allow a "not None" declaration for >>> cdef functions? Obviously, the caller would have to do the check in that >>> case. >> >> Why can't the callee just check it? If it's None, just raise an >> exception like usual? > > It's just that there's a lot more potential for rather easy optimization if > the caller does it. Exactly. The NoneCheckNode is easy to get rid of at any stage in the pipeline, whereas a hard coded None check has a fixed cost at runtime. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 7 May 2012 16:48, Dag Sverre Seljebotn wrote: > On 05/07/2012 03:04 PM, Stefan Behnel wrote: >> >> Dag Sverre Seljebotn, 07.05.2012 13:48: >> >>> Here you go: >>> >>> def foo(np.ndarray[double] a, np.ndarray[double] out=None): >>> if out is None: >>> out = np.empty_like(a) >> >> >> Ah, right - output arguments. Hadn't thought of those. >> >> Still, since you pass None explicitly as a default argument, this code >> wouldn't be impacted by disallowing None for buffers by default. That case >> is already handled specially in the compiler. But a better default would >> prevent the *first* argument from being None. >> >> So, basically, it would do the right thing straight away in your case and >> generate safer and more efficient code for it, whereas now you have to >> test >> 'a' for being None explicitly and Cython won't understand that hint due to >> insufficient static analysis. At least, since my last commit you can make >> Cython do the same thing by declaring it "not None". > > > Yes, thanks! > > > It's really no different from cdef classes. I find it at least a bit more surprising because a buffer unpacking argument is a rather strong hint that you expect something that supports this protocol. The fact that you type your function argument with it hints at the intention to properly unpack it on entry. I'm sure there are lots of users who were or will be surprised when they realise that that doesn't exclude None values. >>> >>> >>> Whereas I think there would be more users surprised by the opposite. >> >> >> We've had enough complaints from users about None being allowed for typed >> arguments already to consider it at least a gotcha of the language. >> >> The main reason we didn't change this behaviour back then was that it >> would >> clearly break user code and we thought we could do without that. That's >> different from considering it "right" and "good". >> >> >> And I remember that we wanted to change the default settings for >> extension >> type arguments from "or None" to "not None" years ago but never >> actually >> did it. > > > I remember that there was such a debate, but I certainly don't remember > that this was the conclusion :-) Maybe not, yes. > I didn't agree with that view then and > I don't now. I don't remember what Robert's view was... > > As far as I can remember (which might be biased towards my personal > view), the conclusion was that we left the current semantics in place, > relying on better control flow analysis to make None-checks cheaper, > and > when those are cheap enough, make the nonecheck directive default to > True At least for buffer arguments, it silently corrupts data or segfaults in the current state of affairs, as you pointed out. Not exactly ideal. >>> >>> >>> No different than writing to a field in a cdef class... >> >> >> Hmm, aren't those None checked? At least cdef method calls are AFAIR. > > > Not at all. That's my whole point -- currently, the rule for None in Cython > is "it's your responsibility to never do a native operation on None". > > I don't like that either, but that's just inherited from Pyrex (and many > projects would get speed regressions etc.). > > I'm not against changing that to "we safely None-check", if done nicely -- > it's just that that should be done everywhere at once. > > In current master (and as far back as I can remember), this code: > > cdef class A: > cdef int field > cdef int method(self): > print self.field > def f(): > cdef A a = None > a.field = 3 > a.method() > > Turns into: > > __pyx_v_a = ((struct __pyx_obj_5test2_A *)Py_None); > __pyx_v_a->field = 3; > ((struct __pyx_vtabstruct_5test2_A *) > __pyx_v_a->__pyx_vtab)->method(__pyx_v_a); > > > > >> I think we should really get back to the habit of making code safe first >> and fast afterwards. > > > Nobody has argued otherwise for some time (since the cdivision thread I > believe), this is all about Pyrex legacy. Guess part of the story is that > there's lots of performance-sensitive code in SAGE using cdef classes which > was written in Pyrex before Cython was around... > > In fact, the nonecheck directive was written by yours truly! And I argued > for making it the default at the time! > > >> Because it uses syntax that is expected to unpack the buffer. If that >> buffer doesn't exist, I'd expect an error. It's like using interfaces: I >> want something here that implements the buffer interface. If it doesn't - >> reject it. >> >> Besides, I hope you are aware that your argumentation stands on the (IMHO >> questionable) fact that "np.ndarray" by itself can be None by default. If >> np.ndarray should not be be allowed to be None by default, why should >> np.ndarray[double]? That argument works in both ways. > > > I'm well aware of it... > > Dag > > __
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 7 May 2012 17:12, Stefan Behnel wrote: > Dag Sverre Seljebotn, 07.05.2012 18:07: >> On 05/07/2012 06:04 PM, mark florisson wrote: >>> On 7 May 2012 12:10, Stefan Behnel wrote: BTW, is there a reason why we shouldn't allow a "not None" declaration for cdef functions? Obviously, the caller would have to do the check in that case. >>> >>> Why can't the callee just check it? If it's None, just raise an >>> exception like usual? >> >> It's just that there's a lot more potential for rather easy optimization if >> the caller does it. > > Exactly. The NoneCheckNode is easy to get rid of at any stage in the > pipeline, whereas a hard coded None check has a fixed cost at runtime. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel I see, yes. I expect a pointer comparison to be reasonably insignificant compared to function call overhead, but it would also reduce the code in the instruction cache. If you take the address of the function though, or if you declare it public in a pxd, you probably don't want to do that, as you still want to be safe when called from C. Could do the same trick as in the 'less annotations' CEP though, that would be nice. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 7 May 2012 17:16, mark florisson wrote: > On 7 May 2012 17:12, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 07.05.2012 18:07: >>> On 05/07/2012 06:04 PM, mark florisson wrote: On 7 May 2012 12:10, Stefan Behnel wrote: > BTW, is there a reason why we shouldn't allow a "not None" declaration for > cdef functions? Obviously, the caller would have to do the check in that > case. Why can't the callee just check it? If it's None, just raise an exception like usual? >>> >>> It's just that there's a lot more potential for rather easy optimization if >>> the caller does it. >> >> Exactly. The NoneCheckNode is easy to get rid of at any stage in the >> pipeline, whereas a hard coded None check has a fixed cost at runtime. >> >> Stefan >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > I see, yes. I expect a pointer comparison to be reasonably > insignificant compared to function call overhead, but it would also > reduce the code in the instruction cache. If you take the address of > the function though, or if you declare it public in a pxd, you > probably don't want to do that, as you still want to be safe when > called from C. Could do the same trick as in the 'less annotations' > CEP though, that would be nice. ... or you could document that 'not None' means the caller cannot pass it in, but that would be weird as you could do it from Cython and get an exception, but not from C :) That would be better specified in the documentation of the function as its contract or whatever. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 0.17
On 7 May 2012 17:04, Vitja Makarov wrote: > 2012/5/7 mark florisson : >> On 6 May 2012 20:41, Matthew Brett wrote: >>> Hi, >>> >>> On Sun, May 6, 2012 at 7:28 AM, mark florisson >>> wrote: Hey, I think we already have quite a bit of functionality (nearly) ready, after merging some pending pull requests maybe it will be a good time for a 0.17 release? I think it would be good to also document to what extent pypy support works, what works and what doesn't. Stefan, since you added a large majority of the features, would you want to be the release manager? In summary, the following pull requests should likely go in - array.array support (unless further discussion prevents that) - fused types runtime buffer dispatch - newaxis - more? The memoryview documentation should also be reworked a bit. Matthew, are you still willing to have a go at that? Otherwise I can clean up the mess first, some things are no longer true and simply outdated, and then have a second opinion. >>> >>> Yes, sorry, I have been taken up by releasing my own project. What's >>> the deadline do you think? I have another big release to do for the >>> end of next week, but I might be able to carve out some time, >>> >>> See you, >>> >>> Matthew >> >> Great, I'd say we're probably not going to release anything within the >> next two weeks, so take your time, there is no hurry really :). > > Hmm, it seems to me that master is currently broken: > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py27-ext/ > > -- > vitja. > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel Quite broken, in fact :) It doesn't ever print error messages property anymore. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 05/07/2012 06:18 PM, mark florisson wrote: On 7 May 2012 17:16, mark florisson wrote: On 7 May 2012 17:12, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 18:07: On 05/07/2012 06:04 PM, mark florisson wrote: On 7 May 2012 12:10, Stefan Behnel wrote: BTW, is there a reason why we shouldn't allow a "not None" declaration for cdef functions? Obviously, the caller would have to do the check in that case. Why can't the callee just check it? If it's None, just raise an exception like usual? It's just that there's a lot more potential for rather easy optimization if the caller does it. Exactly. The NoneCheckNode is easy to get rid of at any stage in the pipeline, whereas a hard coded None check has a fixed cost at runtime. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel I see, yes. I expect a pointer comparison to be reasonably insignificant compared to function call overhead, but it would also reduce the code in the instruction cache. If you take the address of the function though, or if you declare it public in a pxd, you probably don't want to do that, as you still want to be safe when called from C. Could do the same trick as in the 'less annotations' CEP though, that would be nice. ... or you could document that 'not None' means the caller cannot pass it in, but that would be weird as you could do it from Cython and get an exception, but not from C :) That would be better specified in the documentation of the function as its contract or whatever. We're going to need a "Cython ABI" at some point anyway. "Caller checks for None" goes in the ABI docs. Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
On 7 May 2012 17:00, Dag Sverre Seljebotn wrote: > On 05/07/2012 04:16 PM, Stefan Behnel wrote: >> >> Stefan Behnel, 07.05.2012 15:04: >>> >>> Dag Sverre Seljebotn, 07.05.2012 13:48: BTW, with the coming of memoryviews, me and Mark talked about just deprecating the "mytype[...]" meaning buffers, and rather treat it as np.ndarray, array.array etc. being some sort of "template types". That is, we disallow "object[int]" and require some special declarations in the relevant pxd files. >>> >>> >>> Hmm, yes, it's unfortunate that we have two different types of syntax >>> now, >>> one that declares the item type before the brackets and one that declares >>> it afterwards. >> >> >> I actually think this merits some more discussion. Should we consider the >> buffer interface syntax deprecated and focus on the memory view syntax? > > > I think that's the very-long-term intention. Then again, it may be too early > to really tell yet, we just need to see how the memory views play out in > real life and whether they'll be able to replace np.ndarray[double] among > real users. We don't want to shove things down users throats. > > But the use of the trailing-[] syntax needs some cleaning up. Me and Mark > agreed we'd put this proposal forward when we got around to it: > > - Deprecate the "object[double]" form, where [dtype] can be stuck on any > extension type > > - But, do NOT (for the next year at least) deprecate np.ndarray[double], > array.array[double], etc. Basically, there should be a magic flag in > extension type declarations saying "I can be a buffer". > > For one thing, that is sort of needed to open up things for templated cdef > classes/fused types cdef classes, if that is ever implemented. Deprecating is definitely a good start. I think at least if you only allow two types as buffers it will be at least reasonably clear when one is dealing with fused types or buffers. Basically, I think memoryviews should live up to demands of the users, which would mean there would be no reason to keep the buffer syntax. One thing to do is make memoryviews coerce cheaply back to the original objects if wanted (which is likely). Writting np.asarray(mymemview) is kind of annoying. Also, OT (sorry), but I'm kind of worried about the memoryview ABI. If it changes (and I intend to do so), cython modules compiled with different cython versions will become incompatible if they call each other through pxds. Maybe that should be defined as UB... > The semantic meaning of trailing [] is still sort of like the C++ meaning; > that it templates the argument types (except it's lots of special cases in > the compiler for various things rather than a Turing-complete template > language...) > > Dag > >> >> The words-to-punctuation ratio of the latter may hurt the eyes when >> encountering it unprepared, but at least it doesn't require two type >> names, >> of which the one before the brackets (i.e. "object") is mostly useless. >> (Although it does reflect the notion that we are dealing with an object >> here ...) >> >> Stefan >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] checking for "None" in nogil function
Dag Sverre Seljebotn, 07.05.2012 17:48: > On 05/07/2012 03:04 PM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 07.05.2012 13:48: > As far as I can remember (which might be biased towards my personal > view), the conclusion was that we left the current semantics in place, > relying on better control flow analysis to make None-checks cheaper, and > when those are cheap enough, make the nonecheck directive default to > True At least for buffer arguments, it silently corrupts data or segfaults in the current state of affairs, as you pointed out. Not exactly ideal. >>> >>> No different than writing to a field in a cdef class... >> >> Hmm, aren't those None checked? At least cdef method calls are AFAIR. > > Not at all. That's my whole point -- currently, the rule for None in Cython > is "it's your responsibility to never do a native operation on None". > > I don't like that either, but that's just inherited from Pyrex (and many > projects would get speed regressions etc.). > > I'm not against changing that to "we safely None-check", if done nicely -- > it's just that that should be done everywhere at once. I think that gets both of us back on the same track then. :) > In current master (and as far back as I can remember), this code: > > cdef class A: > cdef int field > cdef int method(self): > print self.field > def f(): > cdef A a = None > a.field = 3 > a.method() > > Turns into: > > __pyx_v_a = ((struct __pyx_obj_5test2_A *)Py_None); > __pyx_v_a->field = 3; > ((struct __pyx_vtabstruct_5test2_A *) > __pyx_v_a->__pyx_vtab)->method(__pyx_v_a); Guess I've just been working on the builtins optimiser too long. There, it's obviously not allowed to inject unprotected code like this automatically. It would be fun if we could eventually get to the point where Cython replaces all of the code in f() with an AttributeError, as a combined effort of control flow analysis and dead code removal. A part of that is already there, i.e. Cython would know that 'a' "may be None" in the last two lines and would thus generate a None check with an AttributeError if we allowed it to do that. It wouldn't know that it's always going to be raised, though, so the dead code removal can't strike. I guess that case is just not important enough to implement. BTW, I recently tried to enable None checks in a couple of places and it broke memory views for some reason that I didn't want to investigate. The main problems really seem to be unknown argument values and the lack of proper exception prediction, e.g. in this case: def add_one_2d(int[:,:] buf): for x in xrange(buf.shape[0]): for y in xrange(buf.shape[1]): buf[x,y] += 1 it's statically obvious that only the first access to .shape (outside of all loops) needs a None check and will raise an AttributeError for None, so the check for the second loop can be eliminated as well as the None check on indexing. >> I think we should really get back to the habit of making code safe first >> and fast afterwards. > > Nobody has argued otherwise for some time (since the cdivision thread I > believe), this is all about Pyrex legacy. Guess part of the story is that > there's lots of performance-sensitive code in SAGE using cdef classes which > was written in Pyrex before Cython was around... > > In fact, the nonecheck directive was written by yours truly! And I argued > for making it the default at the time! I've been working on the None checks (and on removing them) repeatedly, although I didn't remember the particular details of discussing the nonecheck directive. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
mark florisson, 07.05.2012 18:28: > On 7 May 2012 17:00, Dag Sverre Seljebotn wrote: >> On 05/07/2012 04:16 PM, Stefan Behnel wrote: >>> Stefan Behnel, 07.05.2012 15:04: Dag Sverre Seljebotn, 07.05.2012 13:48: > BTW, with the coming of memoryviews, me and Mark talked about just > deprecating the "mytype[...]" meaning buffers, and rather treat it as > np.ndarray, array.array etc. being some sort of "template types". That > is, > we disallow "object[int]" and require some special declarations in the > relevant pxd files. Hmm, yes, it's unfortunate that we have two different types of syntax now, one that declares the item type before the brackets and one that declares it afterwards. >>> >>> I actually think this merits some more discussion. Should we consider the >>> buffer interface syntax deprecated and focus on the memory view syntax? >> >> I think that's the very-long-term intention. Then again, it may be too early >> to really tell yet, we just need to see how the memory views play out in >> real life and whether they'll be able to replace np.ndarray[double] among >> real users. We don't want to shove things down users throats. >> >> But the use of the trailing-[] syntax needs some cleaning up. Me and Mark >> agreed we'd put this proposal forward when we got around to it: >> >> - Deprecate the "object[double]" form, where [dtype] can be stuck on any >> extension type >> >> - But, do NOT (for the next year at least) deprecate np.ndarray[double], >> array.array[double], etc. Basically, there should be a magic flag in >> extension type declarations saying "I can be a buffer". >> >> For one thing, that is sort of needed to open up things for templated cdef >> classes/fused types cdef classes, if that is ever implemented. > > Deprecating is definitely a good start. Then the first step on that road is to rework the documentation so that it pushes users into going for memory views instead of the plain buffer syntax. > I think at least if you only > allow two types as buffers it will be at least reasonably clear when > one is dealing with fused types or buffers. > > Basically, I think memoryviews should live up to demands of the users, > which would mean there would be no reason to keep the buffer syntax. > One thing to do is make memoryviews coerce cheaply back to the > original objects if wanted (which is likely). Writting > np.asarray(mymemview) is kind of annoying. ... and also doesn't do the same thing, I believe. > Also, OT (sorry), but I'm kind of worried about the memoryview ABI. If > it changes (and I intend to do so), cython modules compiled with > different cython versions will become incompatible if they call each > other through pxds. Maybe that should be defined as UB... Would there be a way to only use the plain buffer interface for cross module memory view exchange? That could be an acceptable overhead to pay for ABI independence. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
mark florisson, 07.05.2012 18:18: > On 7 May 2012 17:16, mark florisson wrote: >> On 7 May 2012 17:12, Stefan Behnel wrote: >>> Dag Sverre Seljebotn, 07.05.2012 18:07: On 05/07/2012 06:04 PM, mark florisson wrote: > On 7 May 2012 12:10, Stefan Behnel wrote: >> BTW, is there a reason why we shouldn't allow a "not None" declaration >> for >> cdef functions? Obviously, the caller would have to do the check in that >> case. > > Why can't the callee just check it? If it's None, just raise an > exception like usual? It's just that there's a lot more potential for rather easy optimization if the caller does it. >>> >>> Exactly. The NoneCheckNode is easy to get rid of at any stage in the >>> pipeline, whereas a hard coded None check has a fixed cost at runtime. >> >> I see, yes. I expect a pointer comparison to be reasonably >> insignificant compared to function call overhead, but it would also >> reduce the code in the instruction cache. If you take the address of >> the function though, or if you declare it public in a pxd, you >> probably don't want to do that, as you still want to be safe when >> called from C. Could do the same trick as in the 'less annotations' >> CEP though, that would be nice. > > ... or you could document that 'not None' means the caller cannot pass > it in, but that would be weird as you could do it from Cython and get > an exception, but not from C :) That would be better specified in the > documentation of the function as its contract or whatever. "not None" on a cdef function means what all declarations on cdef functions mean: the caller is responsible for doing the appropriate type conversions and checks. If a function accepts an int32 and the caller puts a float32 on the stack, it's not the fault of the callee. The same applies to extension type arguments and None checks. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 0.17
mark florisson, 07.05.2012 18:19: > On 7 May 2012 17:04, Vitja Makarov wrote: >> Hmm, it seems to me that master is currently broken: >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py27-ext/ >> > Quite broken, in fact :) It doesn't ever print error messages property > anymore. Yes, Robert broke the compiler error processing while trying to fix it up for parallel compilation. https://github.com/cython/cython/commit/5d1fddb87fd68991e7fbc79c469273398638b6ff Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
On 7 May 2012 18:00, Stefan Behnel wrote: > mark florisson, 07.05.2012 18:28: >> On 7 May 2012 17:00, Dag Sverre Seljebotn wrote: >>> On 05/07/2012 04:16 PM, Stefan Behnel wrote: Stefan Behnel, 07.05.2012 15:04: > Dag Sverre Seljebotn, 07.05.2012 13:48: >> BTW, with the coming of memoryviews, me and Mark talked about just >> deprecating the "mytype[...]" meaning buffers, and rather treat it as >> np.ndarray, array.array etc. being some sort of "template types". That >> is, >> we disallow "object[int]" and require some special declarations in the >> relevant pxd files. > > Hmm, yes, it's unfortunate that we have two different types of syntax > now, > one that declares the item type before the brackets and one that declares > it afterwards. I actually think this merits some more discussion. Should we consider the buffer interface syntax deprecated and focus on the memory view syntax? >>> >>> I think that's the very-long-term intention. Then again, it may be too early >>> to really tell yet, we just need to see how the memory views play out in >>> real life and whether they'll be able to replace np.ndarray[double] among >>> real users. We don't want to shove things down users throats. >>> >>> But the use of the trailing-[] syntax needs some cleaning up. Me and Mark >>> agreed we'd put this proposal forward when we got around to it: >>> >>> - Deprecate the "object[double]" form, where [dtype] can be stuck on any >>> extension type >>> >>> - But, do NOT (for the next year at least) deprecate np.ndarray[double], >>> array.array[double], etc. Basically, there should be a magic flag in >>> extension type declarations saying "I can be a buffer". >>> >>> For one thing, that is sort of needed to open up things for templated cdef >>> classes/fused types cdef classes, if that is ever implemented. >> >> Deprecating is definitely a good start. > > Then the first step on that road is to rework the documentation so that it > pushes users into going for memory views instead of the plain buffer syntax. > Well, memoryviews are not yet entirely bug free (although the next release will aim to fix the problems pointed out by users so far), and they also have some other problems. >> I think at least if you only >> allow two types as buffers it will be at least reasonably clear when >> one is dealing with fused types or buffers. >> >> Basically, I think memoryviews should live up to demands of the users, >> which would mean there would be no reason to keep the buffer syntax. >> One thing to do is make memoryviews coerce cheaply back to the >> original objects if wanted (which is likely). Writting >> np.asarray(mymemview) is kind of annoying. > > ... and also doesn't do the same thing, I believe. > > >> Also, OT (sorry), but I'm kind of worried about the memoryview ABI. If >> it changes (and I intend to do so), cython modules compiled with >> different cython versions will become incompatible if they call each >> other through pxds. Maybe that should be defined as UB... > > Would there be a way to only use the plain buffer interface for cross > module memory view exchange? That could be an acceptable overhead to pay > for ABI independence. I want to store extra flags and pointers in there as well, so I don't think that will be enough. It will also be rather annoying and complicate calling code. > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
On 05/07/2012 07:00 PM, Stefan Behnel wrote: mark florisson, 07.05.2012 18:28: On 7 May 2012 17:00, Dag Sverre Seljebotn wrote: On 05/07/2012 04:16 PM, Stefan Behnel wrote: Stefan Behnel, 07.05.2012 15:04: Dag Sverre Seljebotn, 07.05.2012 13:48: BTW, with the coming of memoryviews, me and Mark talked about just deprecating the "mytype[...]" meaning buffers, and rather treat it as np.ndarray, array.array etc. being some sort of "template types". That is, we disallow "object[int]" and require some special declarations in the relevant pxd files. Hmm, yes, it's unfortunate that we have two different types of syntax now, one that declares the item type before the brackets and one that declares it afterwards. I actually think this merits some more discussion. Should we consider the buffer interface syntax deprecated and focus on the memory view syntax? I think that's the very-long-term intention. Then again, it may be too early to really tell yet, we just need to see how the memory views play out in real life and whether they'll be able to replace np.ndarray[double] among real users. We don't want to shove things down users throats. But the use of the trailing-[] syntax needs some cleaning up. Me and Mark agreed we'd put this proposal forward when we got around to it: - Deprecate the "object[double]" form, where [dtype] can be stuck on any extension type - But, do NOT (for the next year at least) deprecate np.ndarray[double], array.array[double], etc. Basically, there should be a magic flag in extension type declarations saying "I can be a buffer". For one thing, that is sort of needed to open up things for templated cdef classes/fused types cdef classes, if that is ever implemented. Deprecating is definitely a good start. Then the first step on that road is to rework the documentation so that it pushes users into going for memory views instead of the plain buffer syntax. -1, premature. Dag I think at least if you only allow two types as buffers it will be at least reasonably clear when one is dealing with fused types or buffers. Basically, I think memoryviews should live up to demands of the users, which would mean there would be no reason to keep the buffer syntax. One thing to do is make memoryviews coerce cheaply back to the original objects if wanted (which is likely). Writting np.asarray(mymemview) is kind of annoying. ... and also doesn't do the same thing, I believe. Also, OT (sorry), but I'm kind of worried about the memoryview ABI. If it changes (and I intend to do so), cython modules compiled with different cython versions will become incompatible if they call each other through pxds. Maybe that should be defined as UB... Would there be a way to only use the plain buffer interface for cross module memory view exchange? That could be an acceptable overhead to pay for ABI independence. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On 7 May 2012 18:06, Stefan Behnel wrote: > mark florisson, 07.05.2012 18:18: >> On 7 May 2012 17:16, mark florisson wrote: >>> On 7 May 2012 17:12, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 18:07: > On 05/07/2012 06:04 PM, mark florisson wrote: >> On 7 May 2012 12:10, Stefan Behnel wrote: >>> BTW, is there a reason why we shouldn't allow a "not None" declaration >>> for >>> cdef functions? Obviously, the caller would have to do the check in that >>> case. >> >> Why can't the callee just check it? If it's None, just raise an >> exception like usual? > > It's just that there's a lot more potential for rather easy optimization > if > the caller does it. Exactly. The NoneCheckNode is easy to get rid of at any stage in the pipeline, whereas a hard coded None check has a fixed cost at runtime. >>> >>> I see, yes. I expect a pointer comparison to be reasonably >>> insignificant compared to function call overhead, but it would also >>> reduce the code in the instruction cache. If you take the address of >>> the function though, or if you declare it public in a pxd, you >>> probably don't want to do that, as you still want to be safe when >>> called from C. Could do the same trick as in the 'less annotations' >>> CEP though, that would be nice. >> >> ... or you could document that 'not None' means the caller cannot pass >> it in, but that would be weird as you could do it from Cython and get >> an exception, but not from C :) That would be better specified in the >> documentation of the function as its contract or whatever. > > "not None" on a cdef function means what all declarations on cdef functions > mean: the caller is responsible for doing the appropriate type conversions > and checks. > > If a function accepts an int32 and the caller puts a float32 on the stack, > it's not the fault of the callee. The same applies to extension type > arguments and None checks. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel Well, 'with gil' makes the callee do something. I would personally expect not None to be enforced at least conceptually in the function itself. In any case, I also think it's really not an important issue, as it's likely pretty uncommon to call it from C. If it does break, it will be easy enough to figure out (unless you accidentally corrupt your memory :) So either solution would be fine with me. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] checking for "None" in nogil function
On 7 May 2012 17:52, Stefan Behnel wrote: > Dag Sverre Seljebotn, 07.05.2012 17:48: >> On 05/07/2012 03:04 PM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn, 07.05.2012 13:48: >> As far as I can remember (which might be biased towards my personal >> view), the conclusion was that we left the current semantics in place, >> relying on better control flow analysis to make None-checks cheaper, and >> when those are cheap enough, make the nonecheck directive default to >> True > > At least for buffer arguments, it silently corrupts data or segfaults in > the current state of affairs, as you pointed out. Not exactly ideal. No different than writing to a field in a cdef class... >>> >>> Hmm, aren't those None checked? At least cdef method calls are AFAIR. >> >> Not at all. That's my whole point -- currently, the rule for None in Cython >> is "it's your responsibility to never do a native operation on None". >> >> I don't like that either, but that's just inherited from Pyrex (and many >> projects would get speed regressions etc.). >> >> I'm not against changing that to "we safely None-check", if done nicely -- >> it's just that that should be done everywhere at once. > > I think that gets both of us back on the same track then. :) > > >> In current master (and as far back as I can remember), this code: >> >> cdef class A: >> cdef int field >> cdef int method(self): >> print self.field >> def f(): >> cdef A a = None >> a.field = 3 >> a.method() >> >> Turns into: >> >> __pyx_v_a = ((struct __pyx_obj_5test2_A *)Py_None); >> __pyx_v_a->field = 3; >> ((struct __pyx_vtabstruct_5test2_A *) >> __pyx_v_a->__pyx_vtab)->method(__pyx_v_a); > > Guess I've just been working on the builtins optimiser too long. There, > it's obviously not allowed to inject unprotected code like this automatically. > > It would be fun if we could eventually get to the point where Cython > replaces all of the code in f() with an AttributeError, as a combined > effort of control flow analysis and dead code removal. A part of that is > already there, i.e. Cython would know that 'a' "may be None" in the last > two lines and would thus generate a None check with an AttributeError if we > allowed it to do that. It wouldn't know that it's always going to be > raised, though, so the dead code removal can't strike. I guess that case is > just not important enough to implement. > > BTW, I recently tried to enable None checks in a couple of places and it > broke memory views for some reason that I didn't want to investigate. If you do want to implement it, don't hesitate to ask about any memoryview shenanigans a certain person implemented. > The > main problems really seem to be unknown argument values and the lack of > proper exception prediction, e.g. in this case: > > def add_one_2d(int[:,:] buf): > for x in xrange(buf.shape[0]): > for y in xrange(buf.shape[1]): > buf[x,y] += 1 > > it's statically obvious that only the first access to .shape (outside of > all loops) needs a None check and will raise an AttributeError for None, so > the check for the second loop can be eliminated as well as the None check > on indexing. > Yes. This can be generalized to common subexpression elimination, for bounds checking, for nonechecking, even for wraparound. >>> I think we should really get back to the habit of making code safe first >>> and fast afterwards. >> >> Nobody has argued otherwise for some time (since the cdivision thread I >> believe), this is all about Pyrex legacy. Guess part of the story is that >> there's lots of performance-sensitive code in SAGE using cdef classes which >> was written in Pyrex before Cython was around... >> >> In fact, the nonecheck directive was written by yours truly! And I argued >> for making it the default at the time! > > I've been working on the None checks (and on removing them) repeatedly, > although I didn't remember the particular details of discussing the > nonecheck directive. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] checking for "None" in nogil function
On 7 May 2012 18:18, mark florisson wrote: > On 7 May 2012 17:52, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 07.05.2012 17:48: >>> On 05/07/2012 03:04 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 07.05.2012 13:48: >>> As far as I can remember (which might be biased towards my personal >>> view), the conclusion was that we left the current semantics in place, >>> relying on better control flow analysis to make None-checks cheaper, and >>> when those are cheap enough, make the nonecheck directive default to >>> True >> >> At least for buffer arguments, it silently corrupts data or segfaults in >> the current state of affairs, as you pointed out. Not exactly ideal. > > No different than writing to a field in a cdef class... Hmm, aren't those None checked? At least cdef method calls are AFAIR. >>> >>> Not at all. That's my whole point -- currently, the rule for None in Cython >>> is "it's your responsibility to never do a native operation on None". >>> >>> I don't like that either, but that's just inherited from Pyrex (and many >>> projects would get speed regressions etc.). >>> >>> I'm not against changing that to "we safely None-check", if done nicely -- >>> it's just that that should be done everywhere at once. >> >> I think that gets both of us back on the same track then. :) >> >> >>> In current master (and as far back as I can remember), this code: >>> >>> cdef class A: >>> cdef int field >>> cdef int method(self): >>> print self.field >>> def f(): >>> cdef A a = None >>> a.field = 3 >>> a.method() >>> >>> Turns into: >>> >>> __pyx_v_a = ((struct __pyx_obj_5test2_A *)Py_None); >>> __pyx_v_a->field = 3; >>> ((struct __pyx_vtabstruct_5test2_A *) >>> __pyx_v_a->__pyx_vtab)->method(__pyx_v_a); >> >> Guess I've just been working on the builtins optimiser too long. There, >> it's obviously not allowed to inject unprotected code like this >> automatically. >> >> It would be fun if we could eventually get to the point where Cython >> replaces all of the code in f() with an AttributeError, as a combined >> effort of control flow analysis and dead code removal. A part of that is >> already there, i.e. Cython would know that 'a' "may be None" in the last >> two lines and would thus generate a None check with an AttributeError if we >> allowed it to do that. It wouldn't know that it's always going to be >> raised, though, so the dead code removal can't strike. I guess that case is >> just not important enough to implement. >> >> BTW, I recently tried to enable None checks in a couple of places and it >> broke memory views for some reason that I didn't want to investigate. > > If you do want to implement it, don't hesitate to ask about any > memoryview shenanigans a certain person implemented. > >> The >> main problems really seem to be unknown argument values and the lack of >> proper exception prediction, e.g. in this case: >> >> def add_one_2d(int[:,:] buf): >> for x in xrange(buf.shape[0]): >> for y in xrange(buf.shape[1]): >> buf[x,y] += 1 >> >> it's statically obvious that only the first access to .shape (outside of >> all loops) needs a None check and will raise an AttributeError for None, so >> the check for the second loop can be eliminated as well as the None check >> on indexing. >> > > Yes. This can be generalized to common subexpression elimination, for > bounds checking, for nonechecking, even for wraparound. Given the awesome control flow we have now, I don't think implementing SSA is very hard at all. From there it's also not too hard to implement these things. Pulling these things out of loops is slightly harder though, given guards etc, so you need two implementations, one with all checks in there, and one without any checks. You take the checking version when your conditions outside the loop don't match, as you need to raise the (potential) exception at the right point. I think we should really get back to the habit of making code safe first and fast afterwards. >>> >>> Nobody has argued otherwise for some time (since the cdivision thread I >>> believe), this is all about Pyrex legacy. Guess part of the story is that >>> there's lots of performance-sensitive code in SAGE using cdef classes which >>> was written in Pyrex before Cython was around... >>> >>> In fact, the nonecheck directive was written by yours truly! And I argued >>> for making it the default at the time! >> >> I've been working on the None checks (and on removing them) repeatedly, >> although I didn't remember the particular details of discussing the >> nonecheck directive. >> >> Stefan >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
mark florisson wrote: >On 7 May 2012 17:00, Dag Sverre Seljebotn >wrote: >> On 05/07/2012 04:16 PM, Stefan Behnel wrote: >>> >>> Stefan Behnel, 07.05.2012 15:04: Dag Sverre Seljebotn, 07.05.2012 13:48: > > BTW, with the coming of memoryviews, me and Mark talked about just > deprecating the "mytype[...]" meaning buffers, and rather treat it >as > np.ndarray, array.array etc. being some sort of "template types". >That > is, > we disallow "object[int]" and require some special declarations in >the > relevant pxd files. Hmm, yes, it's unfortunate that we have two different types of >syntax now, one that declares the item type before the brackets and one that >declares it afterwards. >>> >>> >>> I actually think this merits some more discussion. Should we >consider the >>> buffer interface syntax deprecated and focus on the memory view >syntax? >> >> >> I think that's the very-long-term intention. Then again, it may be >too early >> to really tell yet, we just need to see how the memory views play out >in >> real life and whether they'll be able to replace np.ndarray[double] >among >> real users. We don't want to shove things down users throats. >> >> But the use of the trailing-[] syntax needs some cleaning up. Me and >Mark >> agreed we'd put this proposal forward when we got around to it: >> >> - Deprecate the "object[double]" form, where [dtype] can be stuck on >any >> extension type >> >> - But, do NOT (for the next year at least) deprecate >np.ndarray[double], >> array.array[double], etc. Basically, there should be a magic flag in >> extension type declarations saying "I can be a buffer". >> >> For one thing, that is sort of needed to open up things for templated >cdef >> classes/fused types cdef classes, if that is ever implemented. > >Deprecating is definitely a good start. I think at least if you only >allow two types as buffers it will be at least reasonably clear when >one is dealing with fused types or buffers. > >Basically, I think memoryviews should live up to demands of the users, >which would mean there would be no reason to keep the buffer syntax. But they are different approaches -- use a different type/API, or just try to speed up parts of NumPy.. >One thing to do is make memoryviews coerce cheaply back to the >original objects if wanted (which is likely). Writting >np.asarray(mymemview) is kind of annoying. > It is going to be very confusing to have type(mymemview), repr(mymemview), and so on come out as NumPy arrays, but not have the full API of NumPy. Unless you auto-convert on getattr to... If you want to eradicate the distinction between the backing array and the memory view and make it transparent, I really suggest you kick back alive np.ndarray (it can exist in some 'unrealized' state with delayed construction after slicing, and so on). Implementation much the same either way, it is all about how it is presented to the user. Something like mymemview.asobject() could work though, and while not much shorter, it would have some polymorphism that np.asarray does not have (based probably on some custom PEP 3118 extension) Dag >Also, OT (sorry), but I'm kind of worried about the memoryview ABI. If >it changes (and I intend to do so), cython modules compiled with >different cython versions will become incompatible if they call each >other through pxds. Maybe that should be defined as UB... > >> The semantic meaning of trailing [] is still sort of like the C++ >meaning; >> that it templates the argument types (except it's lots of special >cases in >> the compiler for various things rather than a Turing-complete >template >> language...) >> >> Dag >> >>> >>> The words-to-punctuation ratio of the latter may hurt the eyes when >>> encountering it unprepared, but at least it doesn't require two type >>> names, >>> of which the one before the brackets (i.e. "object") is mostly >useless. >>> (Although it does reflect the notion that we are dealing with an >object >>> here ...) >>> >>> Stefan >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >> >> >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel >___ >cython-devel mailing list >cython-devel@python.org >http://mail.python.org/mailman/listinfo/cython-devel -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
On 7 May 2012 19:40, Dag Sverre Seljebotn wrote: > > > mark florisson wrote: > >>On 7 May 2012 17:00, Dag Sverre Seljebotn >>wrote: >>> On 05/07/2012 04:16 PM, Stefan Behnel wrote: Stefan Behnel, 07.05.2012 15:04: > > Dag Sverre Seljebotn, 07.05.2012 13:48: >> >> BTW, with the coming of memoryviews, me and Mark talked about just >> deprecating the "mytype[...]" meaning buffers, and rather treat it >>as >> np.ndarray, array.array etc. being some sort of "template types". >>That >> is, >> we disallow "object[int]" and require some special declarations in >>the >> relevant pxd files. > > > Hmm, yes, it's unfortunate that we have two different types of >>syntax > now, > one that declares the item type before the brackets and one that >>declares > it afterwards. I actually think this merits some more discussion. Should we >>consider the buffer interface syntax deprecated and focus on the memory view >>syntax? >>> >>> >>> I think that's the very-long-term intention. Then again, it may be >>too early >>> to really tell yet, we just need to see how the memory views play out >>in >>> real life and whether they'll be able to replace np.ndarray[double] >>among >>> real users. We don't want to shove things down users throats. >>> >>> But the use of the trailing-[] syntax needs some cleaning up. Me and >>Mark >>> agreed we'd put this proposal forward when we got around to it: >>> >>> - Deprecate the "object[double]" form, where [dtype] can be stuck on >>any >>> extension type >>> >>> - But, do NOT (for the next year at least) deprecate >>np.ndarray[double], >>> array.array[double], etc. Basically, there should be a magic flag in >>> extension type declarations saying "I can be a buffer". >>> >>> For one thing, that is sort of needed to open up things for templated >>cdef >>> classes/fused types cdef classes, if that is ever implemented. >> >>Deprecating is definitely a good start. I think at least if you only >>allow two types as buffers it will be at least reasonably clear when >>one is dealing with fused types or buffers. >> >>Basically, I think memoryviews should live up to demands of the users, >>which would mean there would be no reason to keep the buffer syntax. > > But they are different approaches -- use a different type/API, or just try to > speed up parts of NumPy.. > >>One thing to do is make memoryviews coerce cheaply back to the >>original objects if wanted (which is likely). Writting >>np.asarray(mymemview) is kind of annoying. >> > > > It is going to be very confusing to have type(mymemview), repr(mymemview), > and so on come out as NumPy arrays, but not have the full API of NumPy. > Unless you auto-convert on getattr to... Yeah, the idea is as very simple, as you mention, just keep the object around cached, and when you slice construct one lazily. > If you want to eradicate the distinction between the backing array and the > memory view and make it transparent, I really suggest you kick back alive > np.ndarray (it can exist in some 'unrealized' state with delayed construction > after slicing, and so on). Implementation much the same either way, it is all > about how it is presented to the user. You mean the buffer syntax? > Something like mymemview.asobject() could work though, and while not much > shorter, it would have some polymorphism that np.asarray does not have (based > probably on some custom PEP 3118 extension) I was thinking you could allow the user to register a callback, and use that to coerce from a memoryview back to an object (given a memoryview object). For numpy this would be np.asarray, and the implementation is allowed to cache the result (which it will). It may be too magicky though... but it will be convenient. The memoryview will act as a subclass, meaning that any of its methods will override methods of the converted object. > Dag > > > >>Also, OT (sorry), but I'm kind of worried about the memoryview ABI. If >>it changes (and I intend to do so), cython modules compiled with >>different cython versions will become incompatible if they call each >>other through pxds. Maybe that should be defined as UB... >> >>> The semantic meaning of trailing [] is still sort of like the C++ >>meaning; >>> that it templates the argument types (except it's lots of special >>cases in >>> the compiler for various things rather than a Turing-complete >>template >>> language...) >>> >>> Dag >>> The words-to-punctuation ratio of the latter may hurt the eyes when encountering it unprepared, but at least it doesn't require two type names, of which the one before the brackets (i.e. "object") is mostly >>useless. (Although it does reflect the notion that we are dealing with an >>object here ...) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinf
Re: [Cython] buffer syntax vs. memory view syntax
On Mon, May 7, 2012 at 11:40 AM, Dag Sverre Seljebotn wrote: > > mark florisson wrote: > >>On 7 May 2012 17:00, Dag Sverre Seljebotn >>wrote: >>> On 05/07/2012 04:16 PM, Stefan Behnel wrote: Stefan Behnel, 07.05.2012 15:04: > > Dag Sverre Seljebotn, 07.05.2012 13:48: >> >> BTW, with the coming of memoryviews, me and Mark talked about just >> deprecating the "mytype[...]" meaning buffers, and rather treat it >>as >> np.ndarray, array.array etc. being some sort of "template types". >>That >> is, >> we disallow "object[int]" and require some special declarations in >>the >> relevant pxd files. > > > Hmm, yes, it's unfortunate that we have two different types of >>syntax > now, > one that declares the item type before the brackets and one that >>declares > it afterwards. I actually think this merits some more discussion. Should we >>consider the buffer interface syntax deprecated and focus on the memory view >>syntax? >>> >>> >>> I think that's the very-long-term intention. Then again, it may be >>too early >>> to really tell yet, we just need to see how the memory views play out >>in >>> real life and whether they'll be able to replace np.ndarray[double] >>among >>> real users. We don't want to shove things down users throats. >>> >>> But the use of the trailing-[] syntax needs some cleaning up. Me and >>Mark >>> agreed we'd put this proposal forward when we got around to it: >>> >>> - Deprecate the "object[double]" form, where [dtype] can be stuck on >>any >>> extension type >>> >>> - But, do NOT (for the next year at least) deprecate >>np.ndarray[double], >>> array.array[double], etc. Basically, there should be a magic flag in >>> extension type declarations saying "I can be a buffer". >>> >>> For one thing, that is sort of needed to open up things for templated >>cdef >>> classes/fused types cdef classes, if that is ever implemented. >> >>Deprecating is definitely a good start. I think at least if you only >>allow two types as buffers it will be at least reasonably clear when >>one is dealing with fused types or buffers. >> >>Basically, I think memoryviews should live up to demands of the users, >>which would mean there would be no reason to keep the buffer syntax. > > But they are different approaches -- use a different type/API, or just try to > speed up parts of NumPy.. Part of the question here is whether using np.ndarray[...] currently (or will) offer any additional functionality. While we should likely start steering people this direction, especially over object[...], it seems too soon to deprecate the old-style buffer access. >>One thing to do is make memoryviews coerce cheaply back to the >>original objects if wanted (which is likely). Writting >>np.asarray(mymemview) is kind of annoying. >> > > > It is going to be very confusing to have type(mymemview), repr(mymemview), > and so on come out as NumPy arrays, but not have the full API of NumPy. > Unless you auto-convert on getattr to... > > If you want to eradicate the distinction between the backing array and the > memory view and make it transparent, I really suggest you kick back alive > np.ndarray (it can exist in some 'unrealized' state with delayed construction > after slicing, and so on). Implementation much the same either way, it is all > about how it is presented to the user. > > Something like mymemview.asobject() could work though, and while not much > shorter, it would have some polymorphism that np.asarray does not have (based > probably on some custom PEP 3118 extension) I think it's valuable to have a single name refer to both the Python object (on which methods can be called, and a new one might have to be created if there was slicing) and the memory view. In this light, being able to specify something is both a NumPy array (to use some (overlay optimized?) methods on it and a memory view (for fast indexing) without having two different variables can result in much cleaner code (and an easier transition from untyped NumPy). >>Also, OT (sorry), but I'm kind of worried about the memoryview ABI. If >>it changes (and I intend to do so), cython modules compiled with >>different cython versions will become incompatible if they call each >>other through pxds. Maybe that should be defined as UB... >> >>> The semantic meaning of trailing [] is still sort of like the C++ >>meaning; >>> that it templates the argument types (except it's lots of special >>cases in >>> the compiler for various things rather than a Turing-complete >>template >>> language...) >>> >>> Dag >>> The words-to-punctuation ratio of the latter may hurt the eyes when encountering it unprepared, but at least it doesn't require two type names, of which the one before the brackets (i.e. "object") is mostly >>useless. (Although it does reflect the notion that we are dealing with an >>object here ...) Stefan __
Re: [Cython] 0.17
On Mon, May 7, 2012 at 10:08 AM, Stefan Behnel wrote: > mark florisson, 07.05.2012 18:19: >> On 7 May 2012 17:04, Vitja Makarov wrote: >>> Hmm, it seems to me that master is currently broken: >>> >>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py27-ext/ >>> >> Quite broken, in fact :) It doesn't ever print error messages property >> anymore. > > Yes, Robert broke the compiler error processing while trying to fix it up > for parallel compilation. > > https://github.com/cython/cython/commit/5d1fddb87fd68991e7fbc79c469273398638b6ff Argh, I made that change at the last minute when I was removing a couple of debug print statements. I say we wait another week or so at least to see if any new bug reports come in, but prep a release to be cut soon, but holding it up on any new features that have not gone in yet. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On Mon, May 7, 2012 at 3:40 AM, Dag Sverre Seljebotn wrote: > [moving to dev list] > > > On 05/07/2012 11:17 AM, Stefan Behnel wrote: >> >> Dag Sverre Seljebotn, 07.05.2012 10:44: >>> >>> On 05/07/2012 07:48 AM, Stefan Behnel wrote: shaunc, 07.05.2012 07:13: > > The following code: > > cdef int foo( double[:] bar ) nogil: > return bar is None > > causes: "Converting to Python object not allowed without gil" > > However, I was under the impression that: "When comparing a value with > None, > keep in mind that, if x is a Python object, x is None and x is not None > are > very efficient because they translate directly to C pointer > comparisons," > > I guess the problem is that the memoryview is not a python object -- > indeed, this compiles in the form: > > cdef int foo( object bar ) nogil: > > return bar is None > > But this is a bit counterintuitive... do I need to do "with gil" to > check > if a memoryview is None? And in a nogil function, I'm not necessarily > guaranteed that I don't have the gil -- what is the best way ensure I > have > the gil? (Is there a "secret system call" or should I use a try block?) > > It would seem more appropriate (IMHO, of course :)) to allow "bar is > None" > also when bar is a memoryview I wonder why a memory view should be allowed to be None in the first place. Buffer arguments aren't (because they get unpacked on entry), so why should memory views? >>> >>> >>> ? At least when I implemented it, buffers get unpacked but the case of a >>> None buffer is treated specially, and you're fully allowed (and segfault >>> if >>> you [] it). >> >> >> Hmm, ok, maybe I just got confused by the code then. >> >> I think the docs should state that buffer arguments are best used together >> with the "not None" declaration then. > > > I use them with "=None" default values all the time... then do a > None-check manually. > > It's really no different from cdef classes. > > >> And I remember that we wanted to change the default settings for extension >> type arguments from "or None" to "not None" years ago but never actually >> did it. > > > I remember that there was such a debate, but I certainly don't remember > that this was the conclusion :-) I didn't agree with that view then and > I don't now. I don't remember what Robert's view was... > > As far as I can remember (which might be biased towards my personal > view), the conclusion was that we left the current semantics in place, > relying on better control flow analysis to make None-checks cheaper, and > when those are cheap enough, make the nonecheck directive default to > True (Java is sort of prior art that this can indeed be done?). Yes, that was exactly my point of view. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On Mon, May 7, 2012 at 8:48 AM, Dag Sverre Seljebotn wrote: > On 05/07/2012 03:04 PM, Stefan Behnel wrote: >> >> Dag Sverre Seljebotn, 07.05.2012 13:48: >> >>> Here you go: >>> >>> def foo(np.ndarray[double] a, np.ndarray[double] out=None): >>> if out is None: >>> out = np.empty_like(a) >> >> >> Ah, right - output arguments. Hadn't thought of those. >> >> Still, since you pass None explicitly as a default argument, this code >> wouldn't be impacted by disallowing None for buffers by default. That case >> is already handled specially in the compiler. But a better default would >> prevent the *first* argument from being None. >> >> So, basically, it would do the right thing straight away in your case and >> generate safer and more efficient code for it, whereas now you have to >> test >> 'a' for being None explicitly and Cython won't understand that hint due to >> insufficient static analysis. At least, since my last commit you can make >> Cython do the same thing by declaring it "not None". > > > Yes, thanks! > > > It's really no different from cdef classes. I find it at least a bit more surprising because a buffer unpacking argument is a rather strong hint that you expect something that supports this protocol. The fact that you type your function argument with it hints at the intention to properly unpack it on entry. I'm sure there are lots of users who were or will be surprised when they realise that that doesn't exclude None values. >>> >>> >>> Whereas I think there would be more users surprised by the opposite. >> >> >> We've had enough complaints from users about None being allowed for typed >> arguments already to consider it at least a gotcha of the language. >> >> The main reason we didn't change this behaviour back then was that it >> would >> clearly break user code and we thought we could do without that. That's >> different from considering it "right" and "good". >> >> >> And I remember that we wanted to change the default settings for >> extension >> type arguments from "or None" to "not None" years ago but never >> actually >> did it. > > > I remember that there was such a debate, but I certainly don't remember > that this was the conclusion :-) Maybe not, yes. > I didn't agree with that view then and > I don't now. I don't remember what Robert's view was... > > As far as I can remember (which might be biased towards my personal > view), the conclusion was that we left the current semantics in place, > relying on better control flow analysis to make None-checks cheaper, > and > when those are cheap enough, make the nonecheck directive default to > True At least for buffer arguments, it silently corrupts data or segfaults in the current state of affairs, as you pointed out. Not exactly ideal. >>> >>> >>> No different than writing to a field in a cdef class... >> >> >> Hmm, aren't those None checked? At least cdef method calls are AFAIR. > > > Not at all. That's my whole point -- currently, the rule for None in Cython > is "it's your responsibility to never do a native operation on None". > > I don't like that either, but that's just inherited from Pyrex (and many > projects would get speed regressions etc.). > > I'm not against changing that to "we safely None-check", if done nicely -- > it's just that that should be done everywhere at once. > > In current master (and as far back as I can remember), this code: > > cdef class A: > cdef int field > cdef int method(self): > print self.field > def f(): > cdef A a = None > a.field = 3 > a.method() > > Turns into: > > __pyx_v_a = ((struct __pyx_obj_5test2_A *)Py_None); > __pyx_v_a->field = 3; > ((struct __pyx_vtabstruct_5test2_A *) > __pyx_v_a->__pyx_vtab)->method(__pyx_v_a); > > > > >> I think we should really get back to the habit of making code safe first >> and fast afterwards. > > > Nobody has argued otherwise for some time (since the cdivision thread I > believe), this is all about Pyrex legacy. Guess part of the story is that > there's lots of performance-sensitive code in SAGE using cdef classes which > was written in Pyrex before Cython was around... I think there's a difference between making a new feature fast instead of safe, and introducing a (significant) performance regression to add safety to existing code. Also, the proposed change of "or None" is backwards incompatible, and the eventual solution (as far as I understand it) is to switch back to allowing None (for consistency everywhere else they occur) once we have cheap non checks in place. We can't get around the fact that cdef classes might be None, due to attributes (which must be initialized to something initially). Doing a None check on every buffer access in a loop falls into the significant performance regression, but ideally we could pull it out. - Robert ___
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
Stefan Behnel wrote: The main reason we didn't change this behaviour back then was that it would clearly break user code and we thought we could do without that. That's different from considering it "right" and "good". I changed the None-checking behaviour in Pyrex because I *wanted* to break user code. Or rather, I didn't think it would be a bad thing to make people revisit their code and think properly about whether they really wanted to allow None in each case. -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On Mon, May 7, 2012 at 5:05 PM, Greg Ewing wrote: > Stefan Behnel wrote: > >> The main reason we didn't change this behaviour back then was that it >> would >> clearly break user code and we thought we could do without that. That's >> different from considering it "right" and "good". > > I changed the None-checking behaviour in Pyrex because I *wanted* > to break user code. Or rather, I didn't think it would be a > bad thing to make people revisit their code and think properly > about whether they really wanted to allow None in each case. That's great if you have the time, but revisiting half a million lines of code (e.g. Sage) can be quite expensive... especially as a short-term patch for a better long-term solution (mostly optimized away None-checks on access). By bigger issue of why I don't think this is the right long-term solution is that (cp)def foo(ExnClass arg): ... should behave the same as (cp)def foo(arg): cdef ExnClass a = arg I think part of the difference is also how strongly the line is drawn between the compiled and un-compiled portions of the program. Cython blurs the line (more) between "called from Cython" and "called from Python," and only the latter doing None-checking is inconsistent too. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
Greg Ewing, 08.05.2012 02:05: > Stefan Behnel wrote: > >> The main reason we didn't change this behaviour back then was that it would >> clearly break user code and we thought we could do without that. That's >> different from considering it "right" and "good". > > I changed the None-checking behaviour in Pyrex because I *wanted* > to break user code. Or rather, I didn't think it would be a > bad thing to make people revisit their code and think properly > about whether they really wanted to allow None in each case. The problem here is that it's not very likely that people specifically tested their code with None values, especially if they didn't carefully think of it already when writing it. So changing the default to make people think may not result in making them think before their code starts throwing exceptions somewhere in production. And having to revisit a large amount of code at that point may turn out to be rather expensive. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] How do you trigger a jenkins build?
Vitja Makarov, 07.05.2012 17:08: > I've noticed that old one URL hook doesn't work for me now. > > I tried to check "Build when a change is pushed to GitHub" That should work. > and set "Jenkins Hook URL" to > https://sage.math.washington.edu:8091/hudson/github-webhook/ That isn't configured in Jenkins but in your own GitHub repo as a "post receive URL" (admin->service hooks). Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
mark florisson, 07.05.2012 19:13: > On 7 May 2012 18:06, Stefan Behnel wrote: >> mark florisson, 07.05.2012 18:18: >>> On 7 May 2012 17:16, mark florisson wrote: On 7 May 2012 17:12, Stefan Behnel wrote: > Dag Sverre Seljebotn, 07.05.2012 18:07: >> On 05/07/2012 06:04 PM, mark florisson wrote: >>> On 7 May 2012 12:10, Stefan Behnel wrote: BTW, is there a reason why we shouldn't allow a "not None" declaration for cdef functions? Obviously, the caller would have to do the check in that case. >>> >>> Why can't the callee just check it? If it's None, just raise an >>> exception like usual? >> >> It's just that there's a lot more potential for rather easy optimization >> if >> the caller does it. > > Exactly. The NoneCheckNode is easy to get rid of at any stage in the > pipeline, whereas a hard coded None check has a fixed cost at runtime. I see, yes. I expect a pointer comparison to be reasonably insignificant compared to function call overhead, but it would also reduce the code in the instruction cache. If you take the address of the function though, or if you declare it public in a pxd, you probably don't want to do that, as you still want to be safe when called from C. Could do the same trick as in the 'less annotations' CEP though, that would be nice. >>> >>> ... or you could document that 'not None' means the caller cannot pass >>> it in, but that would be weird as you could do it from Cython and get >>> an exception, but not from C :) That would be better specified in the >>> documentation of the function as its contract or whatever. >> >> "not None" on a cdef function means what all declarations on cdef functions >> mean: the caller is responsible for doing the appropriate type conversions >> and checks. >> >> If a function accepts an int32 and the caller puts a float32 on the stack, >> it's not the fault of the callee. The same applies to extension type >> arguments and None checks. > > Well, 'with gil' makes the callee do something. There's two sides to this one. A "with gil" function can be called from nogil code, so, in a way, the "with gil" declaration is only a short hand for a "nogil" declaration with a "with gil" block inside of the function. It's also a historical artefact of the (long) time before we actually had "with gil" blocks, but it's convenient in that it saves a level of indention that would otherwise uselessly cover the whole function. > I would personally > expect not None to be enforced at least conceptually in the function > itself. As usual: in Python functions, yes, in C functions, no. Vitek's Python wrapper split was a good step into a better design here that reflects this separation of concerns. > In any case, I also think it's really not an important issue, > as it's likely pretty uncommon to call it from C. If it does break, it > will be easy enough to figure out (unless you accidentally corrupt > your memory :) So either solution would be fine with me. Good. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] buffer syntax vs. memory view syntax
Dag Sverre Seljebotn, 07.05.2012 19:10: > On 05/07/2012 07:00 PM, Stefan Behnel wrote: >> mark florisson, 07.05.2012 18:28: >>> On 7 May 2012 17:00, Dag Sverre Seljebotn wrote: On 05/07/2012 04:16 PM, Stefan Behnel wrote: > Stefan Behnel, 07.05.2012 15:04: >> Dag Sverre Seljebotn, 07.05.2012 13:48: >>> BTW, with the coming of memoryviews, me and Mark talked about just >>> deprecating the "mytype[...]" meaning buffers, and rather treat it as >>> np.ndarray, array.array etc. being some sort of "template types". That >>> is, >>> we disallow "object[int]" and require some special declarations in the >>> relevant pxd files. >> >> Hmm, yes, it's unfortunate that we have two different types of syntax >> now, >> one that declares the item type before the brackets and one that >> declares it afterwards. > > I actually think this merits some more discussion. Should we consider the > buffer interface syntax deprecated and focus on the memory view syntax? I think that's the very-long-term intention. Then again, it may be too early to really tell yet, we just need to see how the memory views play out in real life and whether they'll be able to replace np.ndarray[double] among real users. We don't want to shove things down users throats. But the use of the trailing-[] syntax needs some cleaning up. Me and Mark agreed we'd put this proposal forward when we got around to it: - Deprecate the "object[double]" form, where [dtype] can be stuck on any extension type - But, do NOT (for the next year at least) deprecate np.ndarray[double], array.array[double], etc. Basically, there should be a magic flag in extension type declarations saying "I can be a buffer". For one thing, that is sort of needed to open up things for templated cdef classes/fused types cdef classes, if that is ever implemented. >>> >>> Deprecating is definitely a good start. >> >> Then the first step on that road is to rework the documentation so that it >> pushes users into going for memory views instead of the plain buffer syntax. > > -1, premature. Ok, fine. Then we should at least put them next to each other in the NumPy docs and explain a) what the differences are and b) which one users should choose for use cases X, Y and Z. The docs should also make it clear that using "np.ndarray" is only useful for making code work with CPython < 2.6 (and maybe some other cases where NumPy's C-API is leveraged internally), but that this declaration has the drawback of making code less versatile, e.g. because it will *not* work with memoryviews and other kinds of buffers but only plain NumPy arrays. Currently, it basically tells people that statically typed NumPy arrays are the only way to get things working. If it's known to be likely that something will become less important or even deprecated at some point in the future, it's best to make users aware by adapting the documentation ASAP, so that less impacted code gets written in the meantime. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Bug report: enumerate does not accept the "start" argument
Python's built-in function 'enumerate' has a lesser-known 2nd argument that
allows the start value of the enumeration to be set. See the python docs
here:
http://docs.python.org/library/functions.html#enumerate
Cython 0.16 doesn't like it, and only allows one argument.
Here is a simple file to reproduce the failure:
for i in enumerate("abc", 1):
> print i
And the resulting output complaint:
Error compiling Cython file:
>
> ...
> for i in enumerate("abc", 1):
> ^
>
> deploy/_working/_cython_test.pyx:1:18: enumerate() takes at most 1 argument
I have requested a trac login to file bugs like this, but the request is
pending (just sent).
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function
On Mon, May 7, 2012 at 9:41 PM, Stefan Behnel wrote: > Greg Ewing, 08.05.2012 02:05: >> Stefan Behnel wrote: >> >>> The main reason we didn't change this behaviour back then was that it would >>> clearly break user code and we thought we could do without that. That's >>> different from considering it "right" and "good". >> >> I changed the None-checking behaviour in Pyrex because I *wanted* >> to break user code. Or rather, I didn't think it would be a >> bad thing to make people revisit their code and think properly >> about whether they really wanted to allow None in each case. > > The problem here is that it's not very likely that people specifically > tested their code with None values, especially if they didn't carefully > think of it already when writing it. > > So changing the default to make people think may not result in making them > think before their code starts throwing exceptions somewhere in production. > And having to revisit a large amount of code at that point may turn out to > be rather expensive. There's also the problem of people (including me) who wrote a lot of code that *does* correctly handle the None case which, with this change, would suddenly start (erroneously) throwing exceptions without all being revisited. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] How do you trigger a jenkins build?
2012/5/8 Stefan Behnel : > Vitja Makarov, 07.05.2012 17:08: >> I've noticed that old one URL hook doesn't work for me now. >> >> I tried to check "Build when a change is pushed to GitHub" > > That should work. > > >> and set "Jenkins Hook URL" to >> https://sage.math.washington.edu:8091/hudson/github-webhook/ > > That isn't configured in Jenkins but in your own GitHub repo as a "post > receive URL" (admin->service hooks). > Thanks! -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
