[Cython] Suggestion of adding working examples to website
One "feature" that matplotlib (Python's 2D plotting library) has which makes it easy to jump into matplotlib is the huge section of working examples: http://matplotlib.sourceforge.net/examples/index.html and http://matplotlib.sourceforge.net/gallery.html . From this, within a couple of days you can get minimally proficient with matplotlib. Having been (and continuing to be) a new user of Cython, I have found the learning curve to be very steep. The documentation online is pretty good (though it could use some work in places). Sometimes all it would take would be some working examples and the documentation would be completely clear. I taught myself the use of matplotlib through the old cut&paste and iterate method. I find that the one thing that is consistently the most challenging about the Cython docs is the lack of distutils setup.py files for more interesting configurations. Without them it requires a certain amount of guessing, playing, and Googling to make sense of how the pieces are supposed to go together. A working examples section could be VERY helpful in this regards. Also, the options for the distutils extensions are not documented at all so far as I can tell. Since the docs were built with Sphinx, it ought to be pretty easy to pull in docstrings if they exist. Just my 2 cents. I would be happy to work with you all to compile some simple examples for common uses - like the numpy convolve example for instance, and the integration example as well. Regards, Ian ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CEP1000: Native dispatch through callables
On 3 May 2012 13:24, Dag Sverre Seljebotn wrote: > I'm afraid I'm going to try to kick this thread alive again. I want us to > have something that Travis can implement in numba and "his" portion of > SciPy, and also that could be used by NumPy devs. > > Since the decisions are rather arbitrary, perhaps we can try to quickly get > to the "+1" stage (or, depending on how things turn out, a tournament > starting with at most one proposal per person). > > > On 04/20/2012 09:30 AM, Robert Bradshaw wrote: >> >> On Thu, Apr 19, 2012 at 6:18 AM, Dag Sverre Seljebotn >> wrote: >>> >>> On 04/19/2012 01:20 PM, Nathaniel Smith wrote: On Thu, Apr 19, 2012 at 11:56 AM, Dag Sverre Seljebotn wrote: > > > I thought of some drawbacks of getfuncptr: > > - Important: Doesn't allow you to actually inspect the supported > signatures, which is needed (or at least convenient) if you want to use > an > FFI library or do some JIT-ing. So an iteration mechanism is still > needed > in > addition, meaning the number of things for the object to implement > grows > a > bit large. Default implementations help -- OTOH there really wasn't a > major > drawback with the table approach as long as JIT's can just replace it? But this is orthogonal to the table vs. getfuncptr discussion. We're assuming that the table might be extended at runtime, which means you can't use it to determine which signatures are supported. So we need some sort of extra interface for the caller and callee to negotiate a type anyway. (I'm intentionally agnostic about whether it makes more sense for the caller or the callee to be doing the iterating... in general type negotiation could be quite complicated, and I don't think we know enough to get that interface right yet.) >>> >>> >>> >>> Hmm. Right. Let's define an explicit goal for the CEP then. >>> >>> What I care about at is getting the spec right enough such that, e.g., >>> NumPy >>> and SciPy, and other (mostly manually written) C extensions with slow >>> development pace, can be forward-compatible with whatever crazy things >>> Cython or Numba does. >>> >>> There's 4 cases: >>> >>> 1) JIT calls JIT (ruled out straight away) >>> >>> 2) JIT calls static: Say that Numba wants to optimize calls to np.sin >>> etc. >>> without special-casing; this seem to require reading a table of static >>> signatures >>> >>> 3) Static calls JIT: This is the case when scipy.integrate routines >>> calls a >>> Numba callback and Numba generates a specialization for the dtype they >>> explicitly needs. This calls for getfuncptr (but perhaps in a form which >>> we >>> can't quite determine yet?). >>> >>> 4) Static calls static: Either table or getfuncptr works. >>> >>> My gut feeling is go for 2) and 4) in this round => table. >> >> >> getfuncptr is really simple and flexible, but I'm with you on both of >> these to points, and the overhead was not trivial. > > > It's interesting to hear you say the overhead was not trivial (that was my > hunch too but I sort of yielded to peer pressure). I think SAGE has some > history with this -- isn't one of the reasons for the "cpdef" vs. "cdef" > split that "cpdef" has the cost of a single lookup for the presence of a > __dict__ on the object, which was an unacceptable penalty for parts of Sage? > That can't have been much more than a 1ns penalty per instance. > > >> Of course we could offer both, i.e. look at the table first, if it's >> not there call getfuncptr if it's non-null, then fall back to "slow" >> call or error. These are all opt-in depending on how hard you want to >> try to optimize things. > > > That's actually exactly what I was envisioning -- in time (with JITs on both > ends) the table could act sort of as a cache for commonly used overloads, > and getfuncptr would access the others more slowly. > > >> As far as keys vs. interning, I'm also tempted to try to have my cake >> and eat it too. Define a space-friendly encoding for signatures and >> require interning for anything that doesn't fit into a single >> sizeof(void*). The fact that this cutoff would vary for 32 vs 64-bit >> would require some care, but could be done with macros in C. If the >> signatures produce non-aligned "pointer" values there won't be any >> collisions, and this way libraries only have to share in the global >> (Python-level?) interning scheme iff they want to expose/use "large" >> signatures. > > > That was the approach I described to Nathaniel as having the "worst features > of both" -- lack of readable gdb dumps of the keys, and having to define an > interning mechanism for use by the 5% cases that don't fit. > > To sum up hat's been said earlier: The only thing that would blow the key > size above 64 bits except very many arguments would be things like > classes/interfaces/vtables. But in that case, reasonable-sized keys for the > vtables can be compute
Re: [Cython] CEP1000: Native dispatch through callables
On 05/05/2012 01:08 PM, mark florisson wrote: On 3 May 2012 13:24, Dag Sverre Seljebotn wrote: I'm afraid I'm going to try to kick this thread alive again. I want us to have something that Travis can implement in numba and "his" portion of SciPy, and also that could be used by NumPy devs. Since the decisions are rather arbitrary, perhaps we can try to quickly get to the "+1" stage (or, depending on how things turn out, a tournament starting with at most one proposal per person). On 04/20/2012 09:30 AM, Robert Bradshaw wrote: On Thu, Apr 19, 2012 at 6:18 AM, Dag Sverre Seljebotn wrote: On 04/19/2012 01:20 PM, Nathaniel Smith wrote: On Thu, Apr 19, 2012 at 11:56 AM, Dag Sverre Seljebotn wrote: I thought of some drawbacks of getfuncptr: - Important: Doesn't allow you to actually inspect the supported signatures, which is needed (or at least convenient) if you want to use an FFI library or do some JIT-ing. So an iteration mechanism is still needed in addition, meaning the number of things for the object to implement grows a bit large. Default implementations help -- OTOH there really wasn't a major drawback with the table approach as long as JIT's can just replace it? But this is orthogonal to the table vs. getfuncptr discussion. We're assuming that the table might be extended at runtime, which means you can't use it to determine which signatures are supported. So we need some sort of extra interface for the caller and callee to negotiate a type anyway. (I'm intentionally agnostic about whether it makes more sense for the caller or the callee to be doing the iterating... in general type negotiation could be quite complicated, and I don't think we know enough to get that interface right yet.) Hmm. Right. Let's define an explicit goal for the CEP then. What I care about at is getting the spec right enough such that, e.g., NumPy and SciPy, and other (mostly manually written) C extensions with slow development pace, can be forward-compatible with whatever crazy things Cython or Numba does. There's 4 cases: 1) JIT calls JIT (ruled out straight away) 2) JIT calls static: Say that Numba wants to optimize calls to np.sin etc. without special-casing; this seem to require reading a table of static signatures 3) Static calls JIT: This is the case when scipy.integrate routines calls a Numba callback and Numba generates a specialization for the dtype they explicitly needs. This calls for getfuncptr (but perhaps in a form which we can't quite determine yet?). 4) Static calls static: Either table or getfuncptr works. My gut feeling is go for 2) and 4) in this round =>table. getfuncptr is really simple and flexible, but I'm with you on both of these to points, and the overhead was not trivial. It's interesting to hear you say the overhead was not trivial (that was my hunch too but I sort of yielded to peer pressure). I think SAGE has some history with this -- isn't one of the reasons for the "cpdef" vs. "cdef" split that "cpdef" has the cost of a single lookup for the presence of a __dict__ on the object, which was an unacceptable penalty for parts of Sage? That can't have been much more than a 1ns penalty per instance. Of course we could offer both, i.e. look at the table first, if it's not there call getfuncptr if it's non-null, then fall back to "slow" call or error. These are all opt-in depending on how hard you want to try to optimize things. That's actually exactly what I was envisioning -- in time (with JITs on both ends) the table could act sort of as a cache for commonly used overloads, and getfuncptr would access the others more slowly. As far as keys vs. interning, I'm also tempted to try to have my cake and eat it too. Define a space-friendly encoding for signatures and require interning for anything that doesn't fit into a single sizeof(void*). The fact that this cutoff would vary for 32 vs 64-bit would require some care, but could be done with macros in C. If the signatures produce non-aligned "pointer" values there won't be any collisions, and this way libraries only have to share in the global (Python-level?) interning scheme iff they want to expose/use "large" signatures. That was the approach I described to Nathaniel as having the "worst features of both" -- lack of readable gdb dumps of the keys, and having to define an interning mechanism for use by the 5% cases that don't fit. To sum up hat's been said earlier: The only thing that would blow the key size above 64 bits except very many arguments would be things like classes/interfaces/vtables. But in that case, reasonable-sized keys for the vtables can be computed (whether by interning, cryptographic hashing, or a GUID like Microsoft COM). So I'm still +1 on my proposal; but I would be happy with an intern-based proposal if somebody bothers to flesh it out a bit (I don't quite know how I'd do it and would get lost in PyObject* vs. char* and cross-language state sharing...). My pr
Re: [Cython] CEP1000: Native dispatch through callables
On 5 May 2012 17:27, Dag Sverre Seljebotn wrote: > On 05/05/2012 01:08 PM, mark florisson wrote: >> >> On 3 May 2012 13:24, Dag Sverre Seljebotn >> wrote: >>> >>> I'm afraid I'm going to try to kick this thread alive again. I want us to >>> have something that Travis can implement in numba and "his" portion of >>> SciPy, and also that could be used by NumPy devs. >>> >>> Since the decisions are rather arbitrary, perhaps we can try to quickly >>> get >>> to the "+1" stage (or, depending on how things turn out, a tournament >>> starting with at most one proposal per person). >>> >>> >>> On 04/20/2012 09:30 AM, Robert Bradshaw wrote: On Thu, Apr 19, 2012 at 6:18 AM, Dag Sverre Seljebotn wrote: > > > On 04/19/2012 01:20 PM, Nathaniel Smith wrote: >> >> >> >> On Thu, Apr 19, 2012 at 11:56 AM, Dag Sverre Seljebotn >> wrote: >>> >>> >>> >>> I thought of some drawbacks of getfuncptr: >>> >>> - Important: Doesn't allow you to actually inspect the supported >>> signatures, which is needed (or at least convenient) if you want to >>> use >>> an >>> FFI library or do some JIT-ing. So an iteration mechanism is still >>> needed >>> in >>> addition, meaning the number of things for the object to implement >>> grows >>> a >>> bit large. Default implementations help -- OTOH there really wasn't a >>> major >>> drawback with the table approach as long as JIT's can just replace >>> it? >> >> >> >> >> But this is orthogonal to the table vs. getfuncptr discussion. We're >> assuming that the table might be extended at runtime, which means you >> can't use it to determine which signatures are supported. So we need >> some sort of extra interface for the caller and callee to negotiate a >> type anyway. (I'm intentionally agnostic about whether it makes more >> sense for the caller or the callee to be doing the iterating... in >> general type negotiation could be quite complicated, and I don't think >> we know enough to get that interface right yet.) > > > > > Hmm. Right. Let's define an explicit goal for the CEP then. > > What I care about at is getting the spec right enough such that, e.g., > NumPy > and SciPy, and other (mostly manually written) C extensions with slow > development pace, can be forward-compatible with whatever crazy things > Cython or Numba does. > > There's 4 cases: > > 1) JIT calls JIT (ruled out straight away) > > 2) JIT calls static: Say that Numba wants to optimize calls to np.sin > etc. > without special-casing; this seem to require reading a table of static > signatures > > 3) Static calls JIT: This is the case when scipy.integrate routines > calls a > Numba callback and Numba generates a specialization for the dtype they > explicitly needs. This calls for getfuncptr (but perhaps in a form > which > we > can't quite determine yet?). > > 4) Static calls static: Either table or getfuncptr works. > > My gut feeling is go for 2) and 4) in this round => table. getfuncptr is really simple and flexible, but I'm with you on both of these to points, and the overhead was not trivial. >>> >>> >>> >>> It's interesting to hear you say the overhead was not trivial (that was >>> my >>> hunch too but I sort of yielded to peer pressure). I think SAGE has some >>> history with this -- isn't one of the reasons for the "cpdef" vs. "cdef" >>> split that "cpdef" has the cost of a single lookup for the presence of a >>> __dict__ on the object, which was an unacceptable penalty for parts of >>> Sage? >>> That can't have been much more than a 1ns penalty per instance. >>> >>> Of course we could offer both, i.e. look at the table first, if it's not there call getfuncptr if it's non-null, then fall back to "slow" call or error. These are all opt-in depending on how hard you want to try to optimize things. >>> >>> >>> >>> That's actually exactly what I was envisioning -- in time (with JITs on >>> both >>> ends) the table could act sort of as a cache for commonly used overloads, >>> and getfuncptr would access the others more slowly. >>> >>> As far as keys vs. interning, I'm also tempted to try to have my cake and eat it too. Define a space-friendly encoding for signatures and require interning for anything that doesn't fit into a single sizeof(void*). The fact that this cutoff would vary for 32 vs 64-bit would require some care, but could be done with macros in C. If the signatures produce non-aligned "pointer" values there won't be any collisions, and this way libraries only have to share in the global (Python-level?) interning scheme iff they want to expose/use "large" signatures. >>> >>> >>> >>> That was the approach I descr
Re: [Cython] Python array support (#113)
> https://github.com/cython/cython/pull/113 This looks ok to me now. There have been objections back when we discussed the initial patch for array.array support, so what do you think about merging this in? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Python array support (#113)
Stefan Behnel, 05.05.2012 21:50: >> https://github.com/cython/cython/pull/113 > > This looks ok to me now. There have been objections back when we discussed > the initial patch for array.array support, so what do you think about > merging this in? One think I'm not sure about is how to deal with the header file. It would be nice to not rely on an external dependency that users need to ship with their code. Moving this into our utility code to write it into the C file would remove that need, but we don't currently have a way to trigger utility code insertion from .pxd files explicitly. Should we special case "cimport cpython.array" for this? Oh, and maybe we should also provide a fused type for the supported array item types to make it easier for users to write generic array code? (Although the mass of types may be overkill for most users...) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
