Re: [Cython] Control flow graph
On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote: > 2011/2/15 Robert Bradshaw : >> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov >> wrote: >>> Hi! >>> >>> In order to implement "reaching definitions" algorithm. >>> I'm now working on control-flow (or data-flow) graph. >>> >>> Here is funny picture made with graphviz ;) >>> >>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >> >> Cool. Any plans on handling exceptions? >> > > Sure, but I don't have much time for this :( > > Linear block inside try...except body should be split by assignments > and each subblock should point to exception handling entry point. Would every possible failing sub-expression have to point to the exception handling point(s)? I suppose it depends on whether you'll be handling more than assignment tracking. > As result I want to have set of possible assignments for each NameNode > position. > > So handling of uninitialized variables, unused, unused results should be easy, > later it may help to implement local variable deletion. And guess that > could help type inference. Ye. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: > This thread is coming over to cython-dev (and the new cython-devel) > from cython-users because it turns out it will probably require > chaning the Cython code. To get everyone who hasn't been following on > cython-users up to speed, here's a summary of what I'm trying to do: > > That's what I was trying to give with this: > > On Wed, Feb 09, 2011 at 12:23:25PM -0500, W. Trevor King wrote: >> I'm wrapping an external C library with Cython, so I have `mylib.pxd`: >> >> cdef extern from 'mylib.h' >> enum: CONST_A >> enum: CONST_B >> ... >> >> where I declare each constant macro from the library's header `mylib.h`: >> >> #define CONST_A 1 >> #define CONST_B 2 >> ... >> >> Now I want to expose those constants in Python, so I have `expose.pyx`: >> >> cimport mylib >> >> CONST_A = mylib.CONST_A >> CONST_B = mylib.CONST_B >> ... >> >> But the last part seems pretty silly. I'd like to do something like >> >> cimport mylib >> import sys >> >> for name in dir(mylib): >> setattr(sys.modules[__name__], name, getattr(mylib, name)) >> >> which compiles fine, but fails to import with... > > Looking into the Cython internals, everything defined in mylib.pxd is > stored as `Entry`s in a `ModuleScope`, and... > > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: >> > What I'm missing is a way to bind the ModuleScope namespace to a name >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >> > name)` will work in expose.pyx. >> >> You have also hit into the thorny issue that .pxd files are used for >> many things. They may be pure C library declarations with no Python >> module backing, they may be declarations of (externally implemented) >> Python modules (such as numpy.pxd), or they may be declarations for >> Cython-implemented modules. > >> > It seems like it would be easier to generate some kind of wrapper >> > class (PxdModule?) for mylib when it is cimported (at compile time), >> > and then further interactions would take care of themselves (at run >> > time). >> >> Would such an object be created anew for every module that cimports >> the declaration file? > > Hmm, That doesn't sound very nice, does it. However, .pxd files > declaring C libraries have no Python-space presence, so that was my > initial idea. > >> I have toyed with the idea of subclassing the module object itself for >> better support of C-level attributes from the Python (and Cython) >> namespaces. > > Sorry, I don't understand "better support of C-level attributes". Can > you give an example? The extern cpdef declarations are an example of this. >> Here's another idea, what if extern blocks could contain cpdef >> declarations, which would automatically generate a Python-level >> wrappers for the declared members (if possible, otherwise an error)? > > Ah, this sounds good! Of the three .pxd roles you list above, > external Python modules (e.g. numpy) and Cython-implemented modules > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. > What's missing is a way to give (where possible) declarations of > external C libraries a Python presence. cpdef fills this hole nicely, > since its whole purpose is to expose Python interfaces to > C-based elements. In the case of external Python modules, I'm not so sure we want to monkey-patch our stuff in (and where would we do it--on the first import of a cimporting module?) > A side effect of this cpdef change would be that now even bare .pxd > files (no matching .pyx) would have a Python presence, Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live. > so You could do > something like > > cimport mylib as mylib_c > import mylib as mylib_py > import sys > > # Access through Python > for name in dir(mylib_py): > setattr(sys.modules[__name__], name, getattr(mylib_py, name)) I think this smells worse than "import *" > # Direct C access > cdef get_a(): > return mylib_c.CONST_A > > Where the Python access would be the new feature, and list all > cpdef-ed stuff. > > However, from Parsing.py
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
Forgot reply-all... didin't we have this discussion before about making that the default for this list as it is by-far the most common desired behavior? On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: > On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: >>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: >>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: >>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name >>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >>> >> > name)` will work in expose.pyx. >>> >> >>> >> You have also hit into the thorny issue that .pxd files are used for >>> >> many things. They may be pure C library declarations with no Python >>> >> module backing, they may be declarations of (externally implemented) >>> >> Python modules (such as numpy.pxd), or they may be declarations for >>> >> Cython-implemented modules. >>> >> >>> >> Here's another idea, what if extern blocks could contain cpdef >>> >> declarations, which would automatically generate a Python-level >>> >> wrappers for the declared members (if possible, otherwise an error)? >>> > >>> > Ah, this sounds good! Of the three .pxd roles you list above, >>> > external Python modules (e.g. numpy) and Cython-implemented modules >>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. >>> > What's missing is a way to give (where possible) declarations of >>> > external C libraries a Python presence. cpdef fills this hole nicely, >>> > since its whole purpose is to expose Python interfaces to >>> > C-based elements. >>> >>> In the case of external Python modules, I'm not so sure we want to >>> monkey-patch our stuff in >> >> I don't think any of the changes we are suggesting would require >> changes to existing code, so .pxd-s with external implementations >> wouldn't be affected unless they brough the changes upon themselves. > > Say, in numpy.pxd, I have > > cdef extern from "...": > cpdef struct obscure_internal_struct: > ... > > Do we add an "obscure_internal_struct" onto the (global) numpy module? > What if it conflicts with a (runtime) name? This is the issue I'm > bringing up. > >>> (and where would we do it--on the first import of a cimporting >>> module?) >> >> Compilation is an issue. I think that .pxd files should be able to be >> cythoned directly, since then they Cython can build any wrappers they >> request. If the file has a matching .pyx file, cythoning either one >> should compile both together, since they'll produce a single Python >> .so module. > > In this case, I think it may make more sense to consider cimporting > stuff from .pyx files if no .pxd file is present. In any case, this is > a big change. I don't like the idea of always creating such a module > (it may be empty, or have name conflicts) nor the idea of > conditionally compiling it (does one have to look at the contents and > really understand Cython to see if a Python shadow will be created?) > >>> > A side effect of this cpdef change would be that now even bare .pxd >>> > files (no matching .pyx) would have a Python presence, >>> >>> Where would it live? Would we just create this module (in essence, >>> acting as if there was an empty .pyx file sitting there as well)? On >>> this note, it may be worth pursuing the idea of a "cython helper" >>> module where common code and objects could live. >> >> I'm not sure exactly what you mean by "cython helper", but this sounds >> like my 'bare .pyx can create a Python .so module idea above. > > I'm thinking of a place to put, e.g. the generator and bind-able > function classes, which are now re-implemented in every module that > uses them. I think there will be more cases like this in the future > rather than less. C-level code could be #included and linked from > "global" stores as well. However, that's somewhat tangential. > >>> > so You could do >>> > something like >>> > >>> > cimport mylib as mylib_c >>> > import mylib as mylib_py >>> > import sys
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote: > On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: >> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >>> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: >>>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: >>>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >>>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: >>>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name >>>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >>>> >> > name)` will work in expose.pyx. >>>> >> >>>> >> You have also hit into the thorny issue that .pxd files are used for >>>> >> many things. They may be pure C library declarations with no Python >>>> >> module backing, they may be declarations of (externally implemented) >>>> >> Python modules (such as numpy.pxd), or they may be declarations for >>>> >> Cython-implemented modules. >>>> >> >>>> >> Here's another idea, what if extern blocks could contain cpdef >>>> >> declarations, which would automatically generate a Python-level >>>> >> wrappers for the declared members (if possible, otherwise an error)? >>>> > >>>> > Ah, this sounds good! Of the three .pxd roles you list above, >>>> > external Python modules (e.g. numpy) and Cython-implemented modules >>>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. >>>> > What's missing is a way to give (where possible) declarations of >>>> > external C libraries a Python presence. cpdef fills this hole nicely, >>>> > since its whole purpose is to expose Python interfaces to >>>> > C-based elements. >>>> >>>> In the case of external Python modules, I'm not so sure we want to >>>> monkey-patch our stuff in >>> >>> I don't think any of the changes we are suggesting would require >>> changes to existing code, so .pxd-s with external implementations >>> wouldn't be affected unless they brough the changes upon themselves. >> >> Say, in numpy.pxd, I have >> >> cdef extern from "...": >> cpdef struct obscure_internal_struct: >> ... >> >> Do we add an "obscure_internal_struct" onto the (global) numpy module? >> What if it conflicts with a (runtime) name? This is the issue I'm >> bringing up. > > Defining a cpdef *and* a non-matching external implementation should > raise a compile-time error. I agree that there is a useful > distinction between external-C-library and external-Python-module .pxd > wrappers. Perhaps your matching blank .py or .pyx file could serve as > a marker that the .pxd file should be inflated into its own full > fledged python module. I'm not even sure how you would go about > adding attributes to the numpy module. When/how would the > Cython-created attributes get added? Yes, this is exactly the issue. > In the external-C-library case, if you define something incompatible > in the matching .pyx or .py file, Cython will be able to see it and > die with an appropriate error, so you can resolve your programming > mistake. That is only if you have a matching .pyx of .py file. Of course, there could be a module by that name in the user's runtime environment that we don't know about, but that currently is not in conflict with a .pxd only file. > If you try to override anything in a .so compiled module at runtime, > you'd get the same kind of error you currently do trying to rebind a > compiled class' method. That's the desired behavior for statically-bound globals, but implementing it is not so trivial. >>>> (and where would we do it--on the first import of a cimporting >>>> module?) >>> >>> Compilation is an issue. I think that .pxd files should be able to be >>> cythoned directly, since then they Cython can build any wrappers they >>> request. If the file has a matching .pyx file, cythoning either one >>> should compile both together, since they'll produce a single Python >>> .so module. >> >> In this case, I think it may make more sense to consider cimporting >> stuff from .pyx files if no .pxd file is present. > > Can you cimport .pyx files that lack matching .pxd files? > >> In any case, this is a big change. I don&#x
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 1:24 AM, Stefan Behnel wrote: >>> A side effect of this cpdef change would be that now even bare .pxd >>> files (no matching .pyx) would have a Python presence, >> >> Where would it live? Would we just create this module (in essence, >> acting as if there was an empty .pyx file sitting there as well)? On >> this note, it may be worth pursuing the idea of a "cython helper" >> module where common code and objects could live. > > I'm not sure exactly what you mean by "cython helper", but this sounds > like my 'bare .pyx can create a Python .so module idea above. I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential. > > If you generate more than one file from a .pyx, including files that are > shared between compiler runs (or even readily built as .so files), you'd > quickly end up in dependency hell. When to regenerate that file? With what > compiler version and config? C/C++? With/out cleanup code? And where to put > it? Are you sure that place will be writable for all compiler runs? And will > the result be guaranteed to be reproducible, regardless of how many compiler > runs there were in between? I'd probably make it an option to the cythonize(...) command, so all the regeneration/dependency information would be computed based on the whole fileset rather than trying to figure that out for each individual module compilation. I'm thinking this would be an optimization, not a requirement. >>> Unions don't really have a Python parallel, >> >> They can be a cdef class wrapping the union type. > > But I would think coercion would be difficult. Unions are usually (in > my limited experience) for "don't worry about the type, just make sure > it fits in X bytes". How would union->Python conversion work? There would be a wrapping type, e.g. cdef class MyUnion: cdef union_type value > > Wouldn't that have to be a pointer to the real thing instead? Why? This would be a lot messier to get right, and structs and unions already have pass-by-value semantics. with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions). This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it. >>> >>> I think that if Cython will automatically generate a wrapper for >>> >>> cdef public int x >>> >>> it should generate a wrapper for >>> >>> cdef struct X: cdef public int x >> >> Or >> >> cdef public struct X: >> int x >> readonly int z >> private int z >> >> I would perhaps say that non-Pythonable non-private members in public >> structs would be a compile error. > > +1, keep it safe at the beginning. > > >>> There really aren't that metatypes in C, so it doesn't seem like a >>> slippery slope to me. Maybe I'm just missing something... >>> > Ok, I think we're pretty much agreed ;). I think that the next step > is to start working on implementations of: > > * Stand alone .pxd -> Python module I'm not sure we're agreed on this one. > > Same from here. To me, that doesn't make much sense for code that wraps a > library. And if it doesn't wrap a library, there isn't much benefit in > writing a stand-alone .pxd in the first place. A .pyx is much more explicit > and obvious in this case. Especially having some .pxd files that generate > .so files and others that don't will make this very ugly. > > I'd prefer adding support for cimporting from .pyx files instead, > potentially with an automated caching generation of corresponding .pxd files > (maybe as ".pxdg" files to make them easier to handle for users). However, > cyclic dependencies would be tricky to handle automatically then. We could put the in the new __pycache__. > * Extending class cdef/cdpef/public/readonly handling to cover enums, > stucts, and possibly unions. This seems like the best first step. > > +1 > > > * I don't know how to handle things like dummy enums (perhaps by > requiring all cdef-ed enums to be named). All enums in C are named. >>> >>> But my Cython declaration (exposing a C `#define CONST_A 1`): >>> >>> cdef extern from 'mylib.h': >>> enum: CONST_A >>> >>> is not a named enum. >> >> Ah, yes. Maybe we require a name (that would only be used in Python >> space). > > ... require it for cpdef enums, you mean? > > OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to > declare external constants correctly. After all, we loose all type > information that way
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: >> > If you try to override anything in a .so compiled module at runtime, >> > you'd get the same kind of error you currently do trying to rebind a >> > compiled class' method. >> >> That's the desired behavior for statically-bound globals, but >> implementing it is not so trivial. > > It's currently implemented for classes. Are modules that different > from classes? > > >>> import types > >>> types.ModuleType.__class__.__mro__ > (, ) > > So you've got your standard __setattr__ to override with an > error-message generator. What is the implementation difficulty? You can't just add a __setattr__ to a module and have it work, it's a class-level slot. And you don't want to modify all module classes. To do this you have to subclass module itself and insert that in place during import. > I am also unclear about the distinction between cdef and cpdef > (perhaps this should be a new thread?). > > cpdef means "I'm declaring something with C and Python interfaces. > The Python interface is a thin wrapper which can be rebound to an > object of any type, leaving the static C interface inaccessible from > Python." No, you can't re-bind cdef class methods. Despite Cython's attempt to homogenize things for the user, extension classes are quite different than "normal" Python classes. This is a Python C/API issue. > cdef [private] means "I'm declaring something that only has a C > interface." > cdef public means "I'm declaring something with C and Python > interfaces backed by C data. Python code can alter the C data." > cdef readonly means "I'm declaring something with C and Python > interfaces backed by C data. Python code cannot alter the C data." cdef means "back this by a C variable" > This seems to be broken in Cython at the module level, since I can > rebind a cdef-ed class but not a cpdef-ed method: Yes, we don't currently control the module's set/getattr. And classes are "public" by default. > $ cat square.pyx > cdef class A (object): > cdef public int value > > cpdef square(self): > return self.value**2 > $ python -c 'import pyximport; pyximport.install(); import xx; > a = xx.A(); a.value = 3; print a.square(); > a.square = lambda self: self.value' > Traceback (most recent call last): > File "", line 3, in > AttributeError: 'square.A' object attribute 'square' is read-only > $ python -c 'import pyximport; pyximport.install(); import square; > square.A = object; print square.A; a = square.A(); a.value = 3' > > Traceback (most recent call last): > File "", line 2, in > AttributeError: 'object' object has no attribute 'value' > > So the cdef-ed A currently has a rebindable Python interface (acting > like a hypothetical cpdef-ed class), but it's square method is not > rebindable (acting like a hypothetical `cdef readonly`-ed method). > >> >>>> (and where would we do it--on the first import of a cimporting >> >>>> module?) >> >>> >> >>> Compilation is an issue. I think that .pxd files should be able to be >> >>> cythoned directly, since then they Cython can build any wrappers they >> >>> request. If the file has a matching .pyx file, cythoning either one >> >>> should compile both together, since they'll produce a single Python >> >>> .so module. >> >> >> >> ... >> > >> > Under the mantra "explicit is better than implicit", we could have >> > users add something like >> > >> > cdef module "modname" >> > >> > to any .pxd files that should be inflated into Python modules. .pxd >> > files without such a tag would receive the current treatment, error on >> > any cpdef, etc. The drawback of this approach is that it makes Cython >> > more complicated, but if both behaviors are reasonable, there's >> > probably no getting around that. >> >> The other drawback is that it subverts the usual filename <-> module >> name convention that one usually expects. > > I've been convinced that the `cimport .pyx file` route is a better way > to go. > > However, the filename <-> module mapping is troublesome for backing > externally-implemented Python modules (e.g. numpy). If you wanted to > write
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 11:35 AM, W. Trevor King wrote: > On Thu, Feb 17, 2011 at 03:53:13PM -0800, Robert Bradshaw wrote: >> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >> > * Extending class cdef/cdpef/public/readonly handling to cover enums, >> > stucts, and possibly unions. >> >> This seems like the best first step. >> >> > Problems with me getting started now: >> > >> > * I don't know where I should start mucking about in the source. >> >> ...The other files to look at would be Parsing.py,.. > > I've got the the Parsing pretty much working, but portions of the test > suite are now failing with things like > > === Expected errors: === > 1:5: function definition in pxd file must be declared 'cdef inline' > 4:5: inline function definition in pxd file cannot be 'public' > 7:5: inline function definition in pxd file cannot be 'api' > > > === Got errors: === > 1:5: function definition in pxd file must be declared 'cdef inline' > 4:12: inline function definition in pxd file cannot be 'public' > 7:5: inline function definition in pxd file cannot be 'api' > > while cythoning tests/errors/e_func_in_pxd_support.pxd: > > cdef foo(): > return 1 > > cdef public inline foo2(): > return 1 > > cdef api inline foo3(): > return 1 > > The current Cython implementation does not consider 'cdef' to be part > of the node code, and my current implementation does not consider any > qualifiers to be part of the node code. > > It's unclear to me what the "right" position is for the start of a > function (or variable, class, ...) should be, or how errors should be > formatted. My initial impression is that Node.pos should point to the > beginning of the def (here 1:1, 4:1, and 7:1), but that positions of > the various qualifiers (cdef, public, inline, api, ...) should be > cached so that the error points to the offending qualifier. For better or for worse, Cython follows the same inane declarator rules as C, so cdef int a, *b, c[5], (*d)(int) is valid. This colors how c function declarations are expressed in the tree. Qualifiers don't really live in the tree and I'm not sure plumbing the "qualifier node" positions around would be worth the added complexity. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 1:06 PM, W. Trevor King wrote: > So should I go ahead and update the expected error messages in my branch? Yes. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: >> >> >> > If you try to override anything in a .so compiled module at runtime, >> >> > you'd get the same kind of error you currently do trying to rebind a >> >> > compiled class' method. >> >> >> >> That's the desired behavior for statically-bound globals, but >> >> implementing it is not so trivial. >> > >> > It's currently implemented for classes. Are modules that different >> > from classes? >> > >> > >>> import types >> > >>> types.ModuleType.__class__.__mro__ >> > (, ) >> > >> > So you've got your standard __setattr__ to override with an >> > error-message generator. What is the implementation difficulty? >> >> You can't just add a __setattr__ to a module and have it work, it's a >> class-level slot. And you don't want to modify all module classes. To >> do this you have to subclass module itself and insert that in place >> during import. > > So annoying but possible? Not particularly critical either way, > though, since you could always say "don't rebind module-level stuff" > in a project's docs, even if you don't enforce that at the Cython > level. It's still something I'd like to do. This could also be used to allow fast access to module globals and easier sharing of declarations. >> > I am also unclear about the distinction between cdef and cpdef >> > (perhaps this should be a new thread?). >> > >> > cpdef means "I'm declaring something with C and Python interfaces. >> > The Python interface is a thin wrapper which can be rebound to an >> > object of any type, leaving the static C interface inaccessible from >> > Python." >> >> No, you can't re-bind cdef class methods. Despite Cython's attempt to >> homogenize things for the user, extension classes are quite different >> than "normal" Python classes. This is a Python C/API issue. > > Do you mean `cpdef class methods`? If so, you're right: > > $ cat square.pyx > cdef class A (object): > cdef public int value > > cpdef square(self): > return self.value**2 > $ python -c 'import pyximport; pyximport.install(); import square; > square.A.square = lambda self: self.value' > Traceback (most recent call last): > File "", line 1, in > TypeError: can't set attributes of built-in/extension type 'square.A' > > You can't override them in instances either: > > $ python -c 'import pyximport; pyximport.install(); import square; > a = square.A(); a.square = lambda self: self.value' > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'square.A' object attribute 'square' is read-only > > But you can override them in subclasses, which is, I suppose, the > point of cpdef for methods. Yes, otherwise they would have been pretty easy to implement :). >> > cdef [private] means "I'm declaring something that only has a C >> > interface." >> > cdef public means "I'm declaring something with C and Python >> > interfaces backed by C data. Python code can alter the C data." >> > cdef readonly means "I'm declaring something with C and Python >> > interfaces backed by C data. Python code cannot alter the C data." >> >> cdef means "back this by a C variable" > > Ah, ok. That makes more sense. > >> > However, the filename <-> module mapping is troublesome for backing >> > externally-implemented Python modules (e.g. numpy). If you wanted to >> > write a .pxd file backing numpy.random, how would you go about getting >> > your module installed in Cython/Includes/numpy/random.pxd or another >> > path that cython would successfully match with `cimport numpy.random`? >> >> Note that extern blocks (by definition) declare where things come from. > > They declare where the .pxd file looks for .h files, but not where > .pyx files look for the .pxd file. Sorry, I should have said extern blocks that make cdef class declarations (such as our numpy.pxd). >> >> > cdef public struct X: >> >> > int x >> >> > read
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >> > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: >> >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: >> >> > However, the filename <-> module mapping is troublesome for backing >> >> > externally-implemented Python modules (e.g. numpy). If you wanted to >> >> > write a .pxd file backing numpy.random, how would you go about getting >> >> > your module installed in Cython/Includes/numpy/random.pxd or another >> >> > path that cython would successfully match with `cimport numpy.random`? >> >> >> >> Note that extern blocks (by definition) declare where things come from. >> > >> > They declare where the .pxd file looks for .h files, but not where >> > .pyx files look for the .pxd file. >> >> Sorry, I should have said extern blocks that make cdef class >> declarations (such as our numpy.pxd). > > It doesn't look like there are cdef class declarations in numpy.pxd: > > cython $ grep class Cython/Includes/numpy.pxd > ctypedef class numpy.dtype [object PyArray_Descr]: > ctypedef extern class numpy.flatiter [object PyArrayIterObject]: > ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: > ctypedef class numpy.ndarray [object PyArrayObject]: > ctypedef extern class numpy.ufunc [object PyUFuncObject]: > > This still doesn't explain how .pxd files specify which external > implemented Python modules they correspond to. "numpy.dtype" is the fully qualified name of the class it's declaring--the module is "numpy." >> >> >> > cdef public struct X: >> >> >> > int x >> >> >> > readonly int z >> >> >> > private int z >> >> >> > >> >> >> > I would perhaps say that non-Pythonable non-private members in public >> >> >> > structs would be a compile error. >> >> >> >> >> >> +1, keep it safe at the beginning. >> >> > >> >> > -1, keep the code clean and the interface consistent ;). I think the >> >> > struct syntax should be identical to the class syntax, with the >> >> > exception that you can't bind methods to structs. That's the only >> >> > real difference between structs and classes, isn't it? >> >> >> >> In C++, the only difference between structs and classes is that struct >> >> members are public by default. (Not saying that C++ is always the >> >> model to follow, but it gives precedent). And structs can have >> >> function members, that's how to do OOP in C. >> > >> > Oh. Even more reason to have identical struct and class handling in >> > Cython ;). >> > >> > It is unclear to me what `cdef public struct` means. I think it >> > should mean "Python bindings can alter this struct's definition", >> > which doesn't make sense. >> >> I think it should mean "this struct is accessible from Python (as X)" > > Wouldn't that be "cdef readonly struct X"? > >> > Shouldn't the syntax for public members be >> > >> > cdef struct X: >> > cdef public: >> > int x >> > readonly int y >> > private int z >> >> -1 on nesting things like this. Rather than make a struct visible from >> Python iff any of its members are, > > A struct is visible from python iff it is declared public or readonly: > > cdef public struct X: > ... > > or > > cdef readonly struct X: > ... > > I don't think the visibility of the struct as a whole should have any > effect over the visibility of its members, so you should be able to > specify member visibility explicitly with per-member granularity (as > you currently can for classes). > > I was assuming that structs would be public by default (like classes), > but that is obviously configurable. > >> I think it makes more sense to put >> the declaration on the struct itself. We could support >> >> cdef public struct X: >> int x # public >> >> cdef readonly struct Y: >> int y # readonly >> >> cdef [private] struct Z: >> int z # p
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 6:32 PM, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote: >> > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >> >> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >> >> > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: >> >> >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King >> >> >> wrote: >> >> >> > However, the filename <-> module mapping is troublesome for backing >> >> >> > externally-implemented Python modules (e.g. numpy). If you wanted to >> >> >> > write a .pxd file backing numpy.random, how would you go about >> >> >> > getting >> >> >> > your module installed in Cython/Includes/numpy/random.pxd or another >> >> >> > path that cython would successfully match with `cimport >> >> >> > numpy.random`? >> >> >> >> >> >> Note that extern blocks (by definition) declare where things come from. >> >> > >> >> > They declare where the .pxd file looks for .h files, but not where >> >> > .pyx files look for the .pxd file. >> >> >> >> Sorry, I should have said extern blocks that make cdef class >> >> declarations (such as our numpy.pxd). >> > >> > It doesn't look like there are cdef class declarations in numpy.pxd: >> > >> > cython $ grep class Cython/Includes/numpy.pxd >> > ctypedef class numpy.dtype [object PyArray_Descr]: >> > ctypedef extern class numpy.flatiter [object PyArrayIterObject]: >> > ctypedef extern class numpy.broadcast [object >> > PyArrayMultiIterObject]: >> > ctypedef class numpy.ndarray [object PyArrayObject]: >> > ctypedef extern class numpy.ufunc [object PyUFuncObject]: >> > >> > This still doesn't explain how .pxd files specify which external >> > implemented Python modules they correspond to. >> >> "numpy.dtype" is the fully qualified name of the class it's >> declaring--the module is "numpy." > > Hmm, that means that > > cimport numpy > a = numpy.numpy.ndarray > > also compiles. As does a = numpy.foo.x.y.z though perhaps that should be a compile-time error. To get the type, you have to write cimport numpy a = numpy.ndarray. > Wouldn't it make more sense to mirror numpy's module > structure when backing it? You do have nested .pxd modules for > cpython, libc, etc. Yes, in fact that's what we do, but that's not required (e.g. you can have these extern declarations in .pyx files) >> >> >> >> > cdef public struct X: >> >> >> >> > int x >> >> >> >> > readonly int z >> >> >> >> > private int z >> >> >> >> > >> >> >> >> > I would perhaps say that non-Pythonable non-private members in >> >> >> >> > public >> >> >> >> > structs would be a compile error. >> >> >> >> >> >> >> >> +1, keep it safe at the beginning. >> >> >> > >> >> >> > -1, keep the code clean and the interface consistent ;). I think the >> >> >> > struct syntax should be identical to the class syntax, with the >> >> >> > exception that you can't bind methods to structs. That's the only >> >> >> > real difference between structs and classes, isn't it? >> >> >> >> >> >> In C++, the only difference between structs and classes is that struct >> >> >> members are public by default. (Not saying that C++ is always the >> >> >> model to follow, but it gives precedent). And structs can have >> >> >> function members, that's how to do OOP in C. >> >> > >> >> > Oh. Even more reason to have identical struct and class handling in >> >> > Cython ;). >> >> > >> >> > It is unclear to me what `cdef public struct` means. I think it >> >> > should mean "Python bindings can alter this struct's definition", >> >> > which doesn't make sense. >> >> >> >> I think it should mean "this struct i
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote: > W. Trevor King, 20.02.2011 00:31: >> >> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >>> >>> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >>>> >>>> It is unclear to me what `cdef public struct` means. I think it >>>> should mean "Python bindings can alter this struct's definition", >>>> which doesn't make sense. >>> >>> I think it should mean "this struct is accessible from Python (as X)" >> >> Wouldn't that be "cdef readonly struct X"? > > The problem here is that "public" is used differently in different contexts > - usually "exported at the C level" in this kind of context, with the quirk > of meaning "modifiable at the Python level" for cdef class attributes. > > The potentially clearer "cpdef" means "overridable at the Python level" as > well as "visible at the Python level", so it doesn't quite match the second > meaning by itself. > > Structs won't be overridable at the Python level, I guess, so cpdef isn't > quite right. The intention here isn't to export them at the C level either, > so "public" isn't right. > > We could tweak the "cpdef" meaning into "cdef thing mapped to the Python > level, and overridable if supported in the given context", which would lead > to broader applicability. Then we could allow That's what I was thinking. > cpdef readonly struct ... > > I think that's a lot clearer than adding to the double meaning of "public". > I would also prefer if Python accessible cdef class attributes were defined > using "cpdef". We could potentially make that the preferred way in the > future? We could allow it, but -1 to disallowing "cdef class" - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Mon, Feb 21, 2011 at 10:26 AM, Stefan Behnel wrote: > Robert Bradshaw, 21.02.2011 19:11: >> >> On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote: >>> >>> W. Trevor King, 20.02.2011 00:31: >>>> >>>> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >>>>> >>>>> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >>>>>> >>>>>> It is unclear to me what `cdef public struct` means. I think it >>>>>> should mean "Python bindings can alter this struct's definition", >>>>>> which doesn't make sense. >>>>> >>>>> I think it should mean "this struct is accessible from Python (as X)" >>>> >>>> Wouldn't that be "cdef readonly struct X"? >>> >>> The problem here is that "public" is used differently in different >>> contexts >>> - usually "exported at the C level" in this kind of context, with the >>> quirk >>> of meaning "modifiable at the Python level" for cdef class attributes. >>> >>> The potentially clearer "cpdef" means "overridable at the Python level" >>> as >>> well as "visible at the Python level", so it doesn't quite match the >>> second >>> meaning by itself. >>> >>> Structs won't be overridable at the Python level, I guess, so cpdef isn't >>> quite right. The intention here isn't to export them at the C level >>> either, >>> so "public" isn't right. >>> >>> We could tweak the "cpdef" meaning into "cdef thing mapped to the Python >>> level, and overridable if supported in the given context", which would >>> lead >>> to broader applicability. Then we could allow >> >> That's what I was thinking. >> >>> cpdef readonly struct ... >>> >>> I think that's a lot clearer than adding to the double meaning of >>> "public". >>> I would also prefer if Python accessible cdef class attributes were >>> defined >>> using "cpdef". We could potentially make that the preferred way in the >>> future? >> >> We could allow it, but -1 to disallowing "cdef class" > > With "preferred way", I was suggesting that we could *deprecate* > > cdef public int x > cdef readonly object y > > for cdef class properties in favour of > > cpdef int x > cpdef readonly object y Oh, you were talking about members. For sure. > and change the documentation accordingly etc., so that at least new users > get used to the new way. The old way would likely continue to be supported > until Cython 2.0 or so, for the holy cow's sake... :) This kind of thing is much easier to fix than, say, fallout from http://trac.cython.org/cython_trac/ticket/654 - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Py_LIMITED_API
I think it's worth looking into, but I'm skeptical of the benefits at this point, especially compared to all the other stuff that needs doing. Specifically, I think we'll add complexity to generate more verbose and less efficient code to stick to this. I also think it might be worth keeping in mind but not yet investing a huge amount of effort into, as it may not yet have hit the sweet spot of stability vs. utility, and it may be a lot of effort to work around functions that they decide really should have been part of the core. That being said, I would be curious how much of the non-limited API we use from Cython. If it looks to be an easy or non-invasive change, I'd love to support it. - Robert On Wed, Feb 23, 2011 at 7:43 AM, Lisandro Dalcin wrote: > Should we try to support Py_LIMITED_API for Py>=3.2? > > -- > Lisandro Dalcin > --- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] (was: Cython .pxd introspection: listing defined constants)
On Thu, Feb 24, 2011 at 5:42 PM, W. Trevor King wrote: > W. Trevor King, 22.02.2011 18:55: >> I've been working on a more explicit parser that removes the >> ambiguity behind the various visibilities. This will help me ensure >> proper impolementation of my cdef-ed enums/structs/..., and make it >> easier to update visibility syntax in the future. Take a look and >> let me know if you think this is a useful direction to take: > > The refactoring continues as I'm moving my new binding classes into > Symtab. This is, of course, leading to lots of changes, but I've been > running the test suite before each commit to make sure I don't go to > far astray. Anything I miss will eventually lead to a better test > suite ;). > > I'm currently writing up new versions of most Scope methods that use > my classes, and I'll replace the original methods once I've updated > all the code that calls them. Next on the list will be the Nodes > themselves, at which point I think I'll be positioned to put in the > `cdef struct` and whatnot that got this whole thing started. > > Since there has been a fair amount of churn, I though I'd ask for some > more feedback on the general direction I'm headed: > > http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;h=refs/heads/cdef-enums-stucts-and-unions On of my primary motivations to moving to github (or similar) was nicely annotated diffs between branches and the ability to do line-by-line comments. If you could also push to your fork there, that'd be great (otherwise I will ;). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote:
> On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote:
>> W. Trevor King, 22.02.2011 18:55:
>> > I've been working on a more explicit parser that removes the ambiguity
>> > behind the various visibilities. This will help me ensure proper
>> > impolementation of my cdef-ed enums/structs/..., and make it easier to
>> > update visibility syntax in the future. Take a look and let me know
>> > if you think this is a useful direction to take:
>>
>> First thing that caught my eyes: I hope you do not intend to leave the
>> logging usage in the parser. This is seriously performance critical code
>> that Cython compiles down to pretty fast C code. Note that you can use a
>> debugger and Python's profiling/tracing interface to find out what's
>> happening, without code impact.
>
> I can strip them out afterwards, but it helps me figure out what I've
> broken if I shift too much around at the same time.
>
> I don't know enough about Python's trace module to know if I can turn
> on tracing only for functions defined in a single module or not, since
> otherwise its hard for me to separate signal from noise.
I think you can filter things after the fact. It would also be pretty
easy to write a utility that (conditionally) decorates all methods if
a flag is set, which we could leave in. (Wouldn't normally be such a
big deal, but this is one of the bottlenecks of compilation.)
>> Some of the log statements span more than one line, which makes it trickier
>> to strip them out with sed&friends (but backing out the initial changeset
>> would likely make it simple enough to remove the rest manually).
>
> Hmm, perhaps I'll condense the logging statements down onto one (long)
> line a piece, that will make it easy to comment/uncomment them with
> sed/emacs/etc. I suppose once Cython can compile the logging module
> we could leave them in with reduced overhead ;).
>
>> Also note that it's best to write runnable tests ("tests/run/"). The
>> tests in "tests/compile/" are only compiled and imported. See the
>> hacking guide in the wiki. I know you're not there yet with your
>> implementation, I'm just mentioning it.
>
> Thanks for the tip.
>
>> CtxAttribute is the wrong name, though. And the class copy
>> implementation gets even more generic than it already was in the
>> Ctx. I'm not a big fan of that, especially not in the parser. For
>> one, it prevents changing the classes into cdef classes, which had
>> long been on my list for Ctx.
>
> An easy, if uglier, workaround would be to prepend attributes with the
> class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity.
> Then the Ctx class could subclass the current CtxAttribute classes
> instead of binding instances of each of them. That way Ctx would keep
> its traditional flat attribute namespace and easy deepcopy, but
> eveyone in Nodes, etc. that will use the attributes would become
> class-name dependent.
I'd be up for flattening this. In particular, changing every
"entry.name" to "entry.python_binding.name" seems to be a lot of churn
and extra verbiage for not much benefit. The only overlap I see is
name and visibility, and keeping name/cname and adding cvisibility
would be preferable to me.
> The CtxAttribute class is, as its docstring says, just a hook for its
> deepcopy method. With an alternative deepcopy implementation,
> CtxAttribute could be replaced with the standard `object`, so don't
> worry too much about its name at this point ;).
You mean shallow copy?
>> CSource: doesn't sound like quite the right name - it does not describe a C
>> source file but information that Cython has about non-Cython things.
>
> It's a container for attributes that describe the presence and
> location of backing C definitions.
>
> * cdef: "Will there be a backing C defintion?
> * extern: "Has someone else already written it?"
> * name/namespace: "What did they call it?"
>
> If you'd rather I called the class something else, I'm certainly
> willing to change it.
It seems a bit odd to me, but if need be we can rename it later.
However, csource and c_binding seem rather redundant to me, but as
mentioned above I think it's better just to flatten it all.
The changes to parsing look decent to me, but admittedly there's a lot
of renaming churn, so I could have missed something.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King wrote: > On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: >> On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: >> > An easy, if uglier, workaround would be to prepend attributes with the >> > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. >> > Then the Ctx class could subclass the current CtxAttribute classes >> > instead of binding instances of each of them. That way Ctx would keep >> > its traditional flat attribute namespace and easy deepcopy, but >> > eveyone in Nodes, etc. that will use the attributes would become >> > class-name dependent. >> >> I'd be up for flattening this. In particular, changing every >> "entry.name" to "entry.python_binding.name" seems to be a lot of churn >> and extra verbiage for not much benefit. The only overlap I see is >> name and visibility, and keeping name/cname and adding cvisibility >> would be preferable to me. > > That works for me, but I see possible ambiguity in cname. From the > "external C code" docs: > > The other way is to use a C name specification to give different > Cython and C names to the C function. Suppose, for example, that > you want to wrap an external function called eject_tomato(). If > you declare it as: > > cdef extern void c_eject_tomato "eject_tomato" (float speed) > > then its name inside the Cython module will be c_eject_tomato, > whereas its name in C will be eject_tomato. > > In this case I was eventually going to use > > c_source.name = eject_tomato > c_source.namespace = ... > c_binding.name = c_eject_tomato > c_binding.namespace = ... > python_binding.name = eject_tomato > > I'm not sure how Cython handles name/cname with this case at the > moment, but it seems like there are two cnames to keep track of. In this case, cname is "eject_tomato" (which actually gets emitted in the C source file to use it) and name (in both Python and Cython namespace) is "c_eject_tomato." You can think of it as name being what the user sees, and cname being what the C compiler sees. >> > The CtxAttribute class is, as its docstring says, just a hook for its >> > deepcopy method. With an alternative deepcopy implementation, >> > CtxAttribute could be replaced with the standard `object`, so don't >> > worry too much about its name at this point ;). >> >> You mean shallow copy? > > I meant deepcopy, since that seems to be the point of Ctx cloning. At > the moment, however, Ctx deep copies only require Binding shallow > copies. If someone crazy wanted to add mutable attrubites to binding > classes, the Binding deepcopy code would have had to be adjusted. > None of this matters though, if we move back to a flat Ctx attribute > space. OK. As an aside, do you know about the copy.deepcopy() method? >> >> CSource: doesn't sound like quite the right name - it does not describe a >> >> C >> >> source file but information that Cython has about non-Cython things. >> > >> > It's a container for attributes that describe the presence and >> > location of backing C definitions. >> > >> > * cdef: "Will there be a backing C defintion? >> > * extern: "Has someone else already written it?" >> > * name/namespace: "What did they call it?" >> > >> > If you'd rather I called the class something else, I'm certainly >> > willing to change it. >> >> It seems a bit odd to me, but if need be we can rename it later. >> However, csource and c_binding seem rather redundant to me, but as >> mentioned above I think it's better just to flatten it all. >> >> The changes to parsing look decent to me, but admittedly there's a lot >> of renaming churn, so I could have missed something. > > I'll go back and revert out the name changes if you'll confirm that > there is only one meaning for cname. Thanks. BTW, I pushed https://github.com/cython/cython/commit/3552114e1b2a21bf2f81f451d50cd48934ea4eb4 which starts to do some of the back-end implementation. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Expected errors of tests/errors/cdef_members_T517.pxd
On Sat, Feb 26, 2011 at 5:08 AM, W. Trevor King wrote: > I'm splitting Symtab visibilities into explicit C and Python > visibilities, but am having trouble reproducing the expected error > messages for cdef_members_T517: > > $ python runtests.py cdef_members_T517 > Python 2.6.6 (r266:84292, Dec 8 2010, 09:53:33) > [GCC 4.4.4] > > Running tests against Cython 0.14.1+ > > === Expected errors: === > 5:24: C attribute of type 'VoidP' cannot be accessed from Python > 5:24: Cannot convert 'VoidP' to Python object > 6:24: C attribute of type 'VoidP' cannot be accessed from Python > 6:24: Cannot convert 'VoidP' to Python object > 6:24: Cannot convert Python object to 'VoidP' > 14:22: C attribute of type 'Foo' cannot be accessed from Python > 14:22: Cannot convert Python object to 'Foo' > > > === Got errors: === > 5:24: C attribute of type 'VoidP' cannot be accessed from Python > 5:24: Cannot convert 'VoidP' to Python object > 6:24: C attribute of type 'VoidP' cannot be accessed from Python > 6:24: Cannot convert 'VoidP' to Python object > 6:24: Cannot convert Python object to 'VoidP' > 14:22: Cannot convert Python object to 'Foo' > > So my code is not raising: > > 14:22: C attribute of type 'Foo' cannot be accessed from Python > > The relevant lines from tests/errors/cdef_members_T517.pxd are: > > $ grep -n ^ tests/errors/cdef_members_T517.pyx > ... > 8:ctypedef struct Foo: > 9: int i > 10: > 11:cdef class Bar: > 12: cdef Foo foo0 > 13: cdef readonly Foo foo2 > 14: cdef public Foo foo1 > 15: pass > ... > > I see two options: > > 1) Foo attributes are readable from Python, in which case the expected > errors should be changed to match mine. Foo attributes are readable from Python, it converts the struct to a tuple. > 2) Foo attributes are not readable from Python, in which case the > expected errors should be extended with > > 13:22: C attribute of type 'Foo' cannot be accessed from Python > > and probably also also > > 14:22: Cannot convert 'Foo' to Python object > > I think (2) is more reasonable, until I get my public Python enums, > stucts, and unions branch working, after which it would switch to (1) > if the Foo struct was declared with readonly (vs. private) Python > visibility. You can't alter the struct definition from Python, so a > public Python visibility doesn't really make sense for the struct > itself. The visibility refers to the assignability of the member from Python, which makes total sense if there is a object -> Foo conversion. I thought I had implemented this, but I guess now. Once we have wrapping classes this makes total sense, so one can do b = Bar() b.foo1 = Foo(i=100) What's less clear is the behavior of b = Bar() b.foo1 = Foo(i=1) b.foo1.i = 100 print b.foo1 saved = b.foo1 del b print saved - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 26, 2011 at 6:14 PM, W. Trevor King wrote: > On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote: >> On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King wrote: >> > On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: >> >> On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: >> >> > An easy, if uglier, workaround would be to prepend attributes with the >> >> > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. >> >> > Then the Ctx class could subclass the current CtxAttribute classes >> >> > instead of binding instances of each of them. That way Ctx would keep >> >> > its traditional flat attribute namespace and easy deepcopy, but >> >> > eveyone in Nodes, etc. that will use the attributes would become >> >> > class-name dependent. >> >> >> >> I'd be up for flattening this. In particular, changing every >> >> "entry.name" to "entry.python_binding.name" seems to be a lot of churn >> >> and extra verbiage for not much benefit. The only overlap I see is >> >> name and visibility, and keeping name/cname and adding cvisibility >> >> would be preferable to me. >> > >> > That works for me, but I see possible ambiguity in cname. From the >> > "external C code" docs: >> > >> > The other way is to use a C name specification to give different >> > Cython and C names to the C function. Suppose, for example, that >> > you want to wrap an external function called eject_tomato(). If >> > you declare it as: >> > >> > cdef extern void c_eject_tomato "eject_tomato" (float speed) >> > >> > then its name inside the Cython module will be c_eject_tomato, >> > whereas its name in C will be eject_tomato. >> > >> > In this case I was eventually going to use >> > >> > c_source.name = eject_tomato >> > c_source.namespace = ... >> > c_binding.name = c_eject_tomato >> > c_binding.namespace = ... >> > python_binding.name = eject_tomato >> > >> > I'm not sure how Cython handles name/cname with this case at the >> > moment, but it seems like there are two cnames to keep track of. >> >> In this case, cname is "eject_tomato" (which actually gets emitted in >> the C source file to use it) and name (in both Python and Cython >> namespace) is "c_eject_tomato." > > What is the "Cython namespace"? Is that what you get when you cimport > something (vs. the Python namespace being what you get when you import > something). Yes, that would be a good way to understand it. > If so, then that sounds like just two names, so > name/cname it is. Anything that exists in the intersection of the Cython and Python namespace has the same name in both places--the user-visible one, thus name suffices for both. (Also, the Cython namespace is a superset of the Python namespace, possibly with a semantically equivalent but more optimized backing for some objects, e.g. dispatching to C methods directly for builtins.) >> >> > The CtxAttribute class is, as its docstring says, just a hook for its >> >> > deepcopy method. With an alternative deepcopy implementation, >> >> > CtxAttribute could be replaced with the standard `object`, so don't >> >> > worry too much about its name at this point ;). >> >> >> >> You mean shallow copy? >> > >> > I meant deepcopy, since that seems to be the point of Ctx cloning. At >> > the moment, however, Ctx deep copies only require Binding shallow >> > copies. If someone crazy wanted to add mutable attrubites to binding >> > classes, the Binding deepcopy code would have had to be adjusted. >> > None of this matters though, if we move back to a flat Ctx attribute >> > space. >> >> OK. As an aside, do you know about the copy.deepcopy() method? > > Yup, but I was trying to avoid pulling in non-Cython dependencies > since you guys seem to have gone through a lot of trouble to make the > Parsing module fast in C. Ah, OK. It's not always obvious without looking what from the standard library is fast and what is not, but this doesn't look particularly optimized. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Expected errors of tests/errors/cdef_members_T517.pxd
On Sat, Feb 26, 2011 at 6:34 PM, W. Trevor King wrote: > On Sat, Feb 26, 2011 at 10:15:38AM -0800, Robert Bradshaw wrote: >> On Sat, Feb 26, 2011 at 5:08 AM, W. Trevor King wrote: >> > The relevant lines from tests/errors/cdef_members_T517.pxd are: >> > >> > $ grep -n ^ tests/errors/cdef_members_T517.pyx >> > ... >> > 8:ctypedef struct Foo: >> > 9: int i >> > 10: >> > 11:cdef class Bar: >> > 12: cdef Foo foo0 >> > 13: cdef readonly Foo foo2 >> > 14: cdef public Foo foo1 >> > 15: pass >> > ... >> >> ... >> >> What's less clear is the behavior of >> >> b = Bar() >> b.foo1 = Foo(i=1) >> b.foo1.i = 100 >> print b.foo1 >> saved = b.foo1 >> del b >> print saved > > Is the problem choosing between: > > 1) Copy struct data into its own PyObject and let Python manage the > memory. > 2) Pass around a pointer to Cython-managed struct data. Yes, exactly. > I suppose I don't really know enough about how Cython handles this yet > to really see the problem or be able to help with a solution ;). Cython doesn't handle this yet--that's what we get to decide. Another option would be to use weakrefs to implement copy-on-owner-destruction, but that has even subtler corner semantics. b = Bar() b.foo1 = Foo(i=1) first = b.foo1 second = b.foo1 first.i = 100 print second del b first.i = 200 print second I'm leaning towards (1) for ease and efficiency of implementation and it's easier to explain in all cases. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] adding support for __dict__ in extension types
On Mon, Feb 28, 2011 at 8:33 AM, Lisandro Dalcin wrote: > Bringing up this old post... > > On 21 June 2010 15:41, Robert Bradshaw wrote: >> On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: >> >>> If we special case a __dict__ attribute in extension types, i.e: >>> >>> cdef class Foo: >>> cdef dict __dict__ >>> >>> and fill type->tp_dictoffset, then we can support __dict__ in >>> extension types. >>> >>> What do you think? >> >> Sounds like a good idea to me. Note that we check tp_dictoffset for >> fast dispatching for cpdef methods (which would be correct as a dict >> lookup *would* be needed if __dict__ is available). >> > > I still have this patch lying around in my disk. I remember Stefan had > some objections. For example, when the user ask for __dict__, a new > dict is unconditionally created (in CPython, type dict are allocated > on-demand). I propose to get this patch pushed now, and optimize > later (however, I really don't know how to safely implement this > optimization). Note there's also the issue of cpdef methods--if the instance has a __dict__ then a dict lookup must be performed for every method call (to make sure it's not overridden). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] lurker thanks ya'll for all the awesome new stuff you're tinkering on
Thanks both of you for the encouragement. Glad Cython is well liked and well used. On Tue, Mar 1, 2011 at 1:42 AM, Marko Loparic wrote: > On 28 February 2011 21:10, Thomas Keller wrote: >> I don't have the time or knowledge to contribute to Cython right now, but I >> just want to thank everybody for the hard >> work and time they've spent on the project. > > +1. I wish you lots of fun for the Cython workshop! > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King wrote: > Previous discussions in this thread [1,2] have discussed the issues > associated with overloading the 'public' keyword. For an example of > the difficulties this causes, see the ugly workaround [3] in my recent > commit [4]. Definately worth fixing this syntax. > > How do syntax changes with deprecations work? The Parsing module > doesn't seem to be setup to handle things like that. It's not. We rarely deprecate syntax, and we should at least give people a chance to move away from the old syntax first to the new first, so we'll have to support both for a bit at least. When we do deprecate the old way of doing things, I think we'll just use Errors.warning(...). In the long run it'd be nice to have a formal grammar for Cython and use a generated parser rather than maintaining this all by hand, but Parsing.py does more than just parsing at the moment. - Robert > [1] http://mail.python.org/pipermail/cython-devel/2011-February/81.html > [2] http://mail.python.org/pipermail/cython-devel/2011-February/86.html > [3] > https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f#L1R2552 > [4] > https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f > > -- > This email may be signed or encrypted with GPG (http://www.gnupg.org). > The GPG signature (if present) will be attached as 'signature.asc'. > For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy > > My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Thu, Mar 3, 2011 at 4:13 AM, W. Trevor King wrote: > On Wed, Mar 02, 2011 at 06:08:12PM -0800, Robert Bradshaw wrote: >> On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King wrote: >> > Previous discussions in this thread [1,2] have discussed the issues >> > associated with overloading the 'public' keyword. For an example of >> > the difficulties this causes, see the ugly workaround [3] in my recent >> > commit [4]. Definately worth fixing this syntax. >> > >> > How do syntax changes with deprecations work? The Parsing module >> > doesn't seem to be setup to handle things like that. >> >> It's not. >> >> We rarely deprecate syntax, and we should at least give people a >> chance to move away from the old syntax first to the new first, so >> we'll have to support both for a bit at least. When we do deprecate >> the old way of doing things, I think we'll just use >> Errors.warning(...). > > But how would you know which syntax version the source file was aiming > for? There should probably be a way to explicitly specify a Cython > syntax version, probably from the command line and/or setup.py file, > e.g. > > pyrex_syntax_version = (, , ) > > as an argument to Cython.Distutils.extension.Extension, which would > select the appropriate parser. The default (if the version was not > explicitly set), would be to use the current parser. With this setup, > projects would only be forced to upgrade if Cython removed support for > their syntax entirely, and they could choose to upgrade whenever was > most convienient for them. > > I think the syntax version's major/minor number should probably match > the Cython version in which they were introduced. Parser fixes and > compatible extensions would bump the number, and you'd > probably want to deprecate buggy versions more quickly. With the single exception of introducing the cpdef keyword, I can't think of a single instance where Cython's syntax has ever been changed in a backwards incompatible way, so we've never needed to worry about syntax versioning. I don't see anything with the new syntax that's forcing us to be backwards incompatible, and even the local ugliness of supporting both for the time being is better than going down the route of supporting multiple parsers. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
On Thu, Mar 3, 2011 at 1:59 AM, Stefan Behnel wrote: > Vitja Makarov, 03.03.2011 10:48: >> >> To share common sources is a good idea, we can also share "code" in >> libcython-.so >> But then we should handle ABI compatibility problems. > > There is little constant code overlap between modules, and putting that into > a separate library would lead to performance regressions (no more inlining > etc.). There are types that would be nice to share, such as the binding C function and generator types. Both common headers and shared object libraries could be produced, shared, managed by the cythonize(...) build command. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] strange bug in starred target
On Thu, Mar 3, 2011 at 11:44 AM, Vitja Makarov wrote: > 2011/3/3 Vitja Makarov : >> 2011/3/3 Vitja Makarov : >>> This doesn't work: >>> def assign(): >>> a, *b = 1,2,3,4,5 >>> return a, b >>> >> import x >> x.assign() >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "x.pyx", line 6, in x.assign (x.c:445) >>> a, *b = 1,2,3,4,5 >>> TypeError: Expected list, got int >>> >>> And this is ok: >>> >>> def assign(): >>> a, *b = 1,2,3,4,5 >>> return a, b >>> *a, b = 1,2,3,4,5 >>> return a, b >>> >> import x >> x.assign() >>> (1, [2, 3, 4, 5]) >>> >>> >> >> I hope problem is somewhere in map_starred_assignment() >> > > That wasn't as simple as patch is... > > Ticket on trac http://trac.cython.org/cython_trac/ticket/664 > Pull request https://github.com/cython/cython/pull/7 Thanks. Merged. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] "__ipow__ has wrong number of arguments" is back here
On Wed, Mar 2, 2011 at 5:47 PM, Lisandro Dalcin wrote: > On 2 March 2011 17:14, Lisandro Dalcin wrote: >> On 2 March 2011 16:18, Vitja Makarov wrote: >>> Hi! >>> >>> I noticed that this error came back again. >>> >>> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ >>> >> >> I've pushed a fix for __ipow__ to take just two arguments, like >> regular Python classes. >> https://github.com/cython/cython/commit/490fc80e6b494f8ed65f78fc2032d856ec75b8ef >> >> But I forgot to update test for ticket #561, now this should fixed: >> https://github.com/cython/cython/commit/829d6485b77b5688671959f840712c01e6886556 >> > > BTW, do you all agree with this change? To summarize, the C signature of ipow takes three arguments, but the third is often/always garbage, so we shouldn't use it? > I expect this change to break some code, particularly in Sage... It'll be a compile time error fixing a potential runtime crash, so I think that's a good move. Unfortunately the sage-build is still broken due to the fixes for http://trac.cython.org/cython_trac/ticket/654 , but I doubt we use ipow much for two reasons: firstly, most of our objects are immutable, and secondly rather than taking a "modulus" as a third argument for power, the way to do this is to power an element of the appropriate ring, e.g. mod(2, 17)*1000. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Google Summer of Code
Google Summer of Code is going on again this year, and though organizations haven't been chosen again, by all indications, it seems likely that the Python foundation will be a strong participant. Under this umbrella, who out there might be interested in mentoring and/or participating as a student? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Differing Cython-0.14.1.tar.gz files
On Sun, Mar 6, 2011 at 10:54 PM, Stefan Behnel wrote: > Ryan Schmidt, 06.03.2011 23:12: >> >> There are two different files called Cython-0.14.1.tar.gz -- one in >> http://www.cython.org/release/ and a different one in >> http://pypi.python.org/packages/source/C/Cython/: > > Intersting. Do you mean "different" as in "different content" (i.e. sources > etc.), or just as in "md5sum gives different hashes"? > > >> Why don't you release a version of your software as a single distfile >> that is identical on all servers? > > Well, I don't think there is a reason for that, simply because I doubt that > it's being done intentionally. I guess it just lacks a process. I just downloaded both tarballs and it looks like there were a small number of commits in the cython.org one that weren't on PyPi. I thought I had uploaded and then done sdist, but maybe I messed something up with branches while doing some last tests. My bad, I've copied the PyPi ones to cython.org. Thanks for bringing this to our attention, we'll be more careful about this in the future. > In any case, I doubt that there are any differences between the tar.gz > files, except for file modification times and potentially the creation time > of the C sources. The build process is deterministic. This is what usually happens, though we should re-download to get exactly the same file. > IMHO, the best way to make the releases would be to run > > setup.py sdist register upload > > to push them to PyPI, and then take the same tar.gz and copy it over to > cython.org. In any case, the one on PyPI should always take the lead, as > that's what people get when they run easy_install. +1. > I also think we should start signing the released archives. This can be done > via distutils' "upload" command by passing > > upload --sign --identity=[e-mail-address-of-key] Good idea. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Sources list handling in Build/Dependencies.py:create_extension_list
On Wed, Mar 9, 2011 at 3:13 AM, Oleksandr Kreshchenko
wrote:
> Hello!
>
> Following the "CEP 201 - Distutils Preprocessing"
> (http://wiki.cython.org/enhancements/distutils_preprocessing)
> there are two possibilities to build a module with cythonize function in
> setup.py.
>
> Second one uses list of distutils.extension.Extension class which accept all
> necessary compiler and
> linker options along with list of non-pyx sources to compile and link with
> for a particular extension module.
> As this way appears more natural for me I do::
>
> ext_modules = cythonize([Extension(
> 'CyCont', ['CyCont.pyx', 'Cont.c', 'ContCet.c', 'Clash.c',
> 'ContInit.c', 'MemMan.c', '../Log/Log.c'],
> define_macros = [('CONT_API', '')], # None value expands to 1,
> whereas '' to empty
> include_dirs = dirs, #libraries = ['libgcov'],
> extra_compile_args = sys_comp_opts[sys.platform],
> extra_link_args = sys_link_opts[sys.platform])],
> show_version = 1, language_level = 3, verbose = 1, include_path = dirs)
>
> But, only CyCont.c is created from CyCont.pyx then compiled and linked into
> the module CyCont.so,
> but the rest of sources to be wrapped are completely ignored in the build
> process.
> So, doing import CyCont I've got undefined symbol error.
>
> However when I move 'CyCont.pyx' to the end of the sources list i.e.::
>
> ext_modules = cythonize([Extension(
> 'CyCont', ['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c',
> 'MemMan.c', '../Log/Log.c', 'CyCont.pyx'],
>
> ...
>
> the module is built properly.
>
> Execution flow goes though Cython/Build/Dependencies.py (latest revision)::
>
> 393 def create_extension_list(patterns, exclude=[], ctx=None, aliases=None)
> ...
> 404 for pattern in patterns:
> 405 if isinstance(pattern, str):
> 406 filepattern = pattern
> 407 template = None
> 408 name = '*'
> 409 base = None
> 410 exn_type = Extension
> 411 elif isinstance(pattern, Extension):
> 412 filepattern = pattern.sources[0]
> 413 if os.path.splitext(filepattern)[1] not in ('.py', '.pyx'):
> 414 # ignore non-cython modules
> 415 module_list.append(pattern)
> 416 continue
> 417 template = pattern
> 418 name = template.name
> 419 base = DistutilsInfo(exn=template)
> 420 exn_type = template.__class__
> ...
> 423 for file in glob(filepattern):
> ...
> 437 module_list.append(exn_type(
> 438 name=module_name,
> 439 sources=[file],
> 440 **kwds))
> ...
> 443 return module_list
>
> If the first list member (sources[0]) is *.pyx or *.py then the only list
> member is CyCont.pyx (see line 439).
>
> Otherwise if CyCont.pyx is placed at the end of the list or, generally, the
> first list member is
> neither *.pyx nor *.py than the loop over patterns/extensions (on line 404)
> breaks at the line 416
> so sources list is left untouched and cythonize function (line 446) does the
> job.
>
> This difference is a bug. Isn't it?
Yes, for sure, thanks for the report.
https://github.com/cython/cython/commit/0fefeb25858ac943a39c4f9ca7e0a493052517a1
> Have "sources[0]" on line 412 and "sources=[file]" on line 439 to be
> revised?
>
>
> BTW, I catch the idea of twiking an extension in setup.py like this::
>
> for ext in ext_modules:
> ext.define_macros = [('CONT_API', '')]
> ext.extra_compile_args = sys_comp_opts[sys.platform]
> ext.extra_link_args = sys_link_opts[sys.platform]
> ext.sources.extend(['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c',
> 'MemMan.c', '../Log/Log.c'])
>
> but it seams **better** to handle this stuff in the first place within
> Extension(...),
> not to mention using module-level distutils directives.
Yep. The build code is set to handle subclasses of Extension, but
there's only so much you can do, otherwise it becomes difficult to use
two projects that need different distutils hacks.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] OpenMP support
On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted wrote: > A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigué: >> mark florisson, 08.03.2011 18:00: >> > What I meant was that the >> > wrapper returned by the decorator would have to call the closure >> > for every iteration, which introduces function call overhead. >> > >> >[...] >> > >> > I guess we just have to establish what we want to do: do we >> > want to support code with Python objects (and exceptions etc), or >> > just C code written in Cython? >> >> I like the approach that Sturla mentioned: using closures to >> implement worker threads. I think that's very pythonic. You could do >> something like this, for example: >> >> def worker(): >> for item in queue: >> with nogil: >> do_stuff(item) >> >> queue.extend(work_items) >> start_threads(worker, count) >> >> Note that the queue is only needed to tell the thread what to work >> on. A lot of things can be shared over the closure. So the queue may >> not even be required in many cases. > > I like this approach too. I suppose that you will need to annotate the > items so that they are not Python objects, no? Something like: > > def worker(): > cdef int item # tell that item is not a Python object! > for item in queue: > with nogil: > do_stuff(item) > > queue.extend(work_items) > start_threads(worker, count) On a slightly higher level, are we just trying to use OpenMP from Cython, or are we trying to build it into the language? If the former, it may make sense to stick closer than one might otherwise be tempted in terms of API to the underlying C to leverage the existing documentation. A library with a more Pythonic interface could perhaps be written on top of that. Alternatively, if we're building it into Cython itself, I'd it might be worth modeling it after the multiprocessing module (though I understand it would be implemented with threads), which I think is a decent enough model for managing embarrassingly parallel operations. The above code is similar to that, though I'd prefer the for loop implicit rather than as part of the worker method (or at least as an argument). If we went this route, what are the advantages of using OpenMP over, say, pthreads in the background? (And could the latter be done with just a library + some fancy GIL specifications?) One thing that's nice about OpenMP as implemented in C is that the serial code looks almost exactly like the parallel code; the code at http://wiki.cython.org/enhancements/openmp has this property too. Also, I like the idea of being able to hold the GIL by the invoking thread and having the "sharing" threads do the appropriate locking among themselves when needed if possible, e.g. for exception raising. Another thought I had is, there might be other usecases for being able to emit generic pragmas statements, how far would that get us? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Fri, Mar 11, 2011 at 9:36 AM, Stefan Behnel wrote: > Stefan Behnel, 11.03.2011 15:08: >> >> Vitja Makarov, 11.03.2011 15:04: >>> >>> 2011/3/11 Stefan Behnel: Personally, I think it would be nice to keep up Python's semantics, but when I implemented this I broke quite some code in Sage (you may have noticed that the sage-build project in Hudson has been red for a while). There are things in C and especially in C++ that cannot be easily copied into a temporary variable in order to make sure they are evaluated before the following arguments. This is not a problem for Python function calls where all arguments end up being copied (and often converted) anyway. It is a problem for C function calls, though. >>> f(g(a), a.x, h(a)) >>> >>> Why could not this be translated into: >>> >>> tmp1 = g(a) >>> tmp2 = a.x >>> tmp3 = h(a) >>> >>> f(tmp1, tmp2, tmp3) >> >> See above. > > To be a little clearer here, it's a problem in C for example with struct > values. Copying them by value into a temp variable can be expensive, > potentially twice as expensive as simply passing them into the function > normally. Yep, and some types (e.g. array types) can't be assigned to at all. FWIW, the issues with Sage is that many libraries use the "stack allocated, pass-by-reference" trick typedef foo_struct foo_t[1] but we have declared them to be of type "void*" because we don't care about or want to muck with the internals. Cleaning this up is something we should do, but is taking a while, and aside from that it makes Cython even more dependent on correct type declarations (and backwards incompatible in this regard). Sage is a great test for the buildbot, so keeping it red for so long is not a good thing either. > Not sure what kind of additional devilry C++ provides here, but I'd expect > that object values can exhibit bizarre behaviour when being assigned. Maybe > others can enlighten me here. Yes, C++ allows overloading of the assignment operator, so assigning may lead to arbitrary code (as well as probably an expensive copy, as with structs, and structs in C++ are really just classes with a different visibility). > I have no idea how many cases there actually are that we can't handle or > that may lead to a performance degradation when using temps, but the problem > is that they exist at all. Note that this applies not just to function arguments, but a host of other places, e.g. the order of evaluation of "f() + g()" in C is unspecified. Fleshing this out completely will lead to a lot more temps and verbose C code. And then we'd just cross our fingers that the C compiler was able to optimize all these temps away (though still possibly producing inferior code if it were able to switch order of execution). Whatever we do, we need a flag/directive. Perhaps there's a way to guarantee correct ordering for all valid Python code, even in the presence of (function and variable) type inference, but allow some leeway for explicitly declared C functions. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: > Robert Bradshaw, 11.03.2011 19:33: >> >> On Fri, Mar 11, 2011 at 9:36 AM, Stefan Behnel >> wrote: >>> >>> Stefan Behnel, 11.03.2011 15:08: >>>> >>>> Vitja Makarov, 11.03.2011 15:04: >>>>> >>>>> 2011/3/11 Stefan Behnel: >>>>>> >>>>>> Personally, I think it would be nice to keep up Python's semantics, >>>>>> but >>>>>> when >>>>>> I implemented this I broke quite some code in Sage (you may have >>>>>> noticed >>>>>> that the sage-build project in Hudson has been red for a while). There >>>>>> are >>>>>> things in C and especially in C++ that cannot be easily copied into a >>>>>> temporary variable in order to make sure they are evaluated before the >>>>>> following arguments. This is not a problem for Python function calls >>>>>> where >>>>>> all arguments end up being copied (and often converted) anyway. It is >>>>>> a >>>>>> problem for C function calls, though. >>>>> >>>>>> f(g(a), a.x, h(a)) >>>>> >>>>> Why could not this be translated into: >>>>> >>>>> tmp1 = g(a) >>>>> tmp2 = a.x >>>>> tmp3 = h(a) >>>>> >>>>> f(tmp1, tmp2, tmp3) >>>> >>>> See above. >>> >>> To be a little clearer here, it's a problem in C for example with struct >>> values. Copying them by value into a temp variable can be expensive, >>> potentially twice as expensive as simply passing them into the function >>> normally. >> >> Yep, and some types (e.g. array types) can't be assigned to at all. >> FWIW, the issues with Sage is that many libraries use the "stack >> allocated, pass-by-reference" trick >> >> typedef foo_struct foo_t[1] >> >> but we have declared them to be of type "void*" because we don't care >> about or want to muck with the internals. Cleaning this up is >> something we should do, but is taking a while, and aside from that it >> makes Cython even more dependent on correct type declarations (and >> backwards incompatible in this regard). Sage is a great test for the >> buildbot, so keeping it red for so long is not a good thing either. >> >>> Not sure what kind of additional devilry C++ provides here, but I'd >>> expect >>> that object values can exhibit bizarre behaviour when being assigned. >>> Maybe >>> others can enlighten me here. >> >> Yes, C++ allows overloading of the assignment operator, so assigning >> may lead to arbitrary code (as well as probably an expensive copy, as >> with structs, and structs in C++ are really just classes with a >> different visibility). >> >>> I have no idea how many cases there actually are that we can't handle or >>> that may lead to a performance degradation when using temps, but the >>> problem >>> is that they exist at all. >> >> Note that this applies not just to function arguments, but a host of >> other places, e.g. the order of evaluation of "f() + g()" in C is >> unspecified. Fleshing this out completely will lead to a lot more >> temps and verbose C code. And then we'd just cross our fingers that >> the C compiler was able to optimize all these temps away (though still >> possibly producing inferior code if it were able to switch order of >> execution). >> >> Whatever we do, we need a flag/directive. Perhaps there's a way to >> guarantee correct ordering for all valid Python code > > Plain Python code should never suffer from the original problem (with the > obvious exception of cpdef functions), but I also see how type inference can > affect this guarantee for the evaluation order in expressions. In particular, we need to be careful if we ever automatically cpdef a function (as part of a future optimization). > I agree with Greg, though, that it has a code smell if (multiple) > subexpressions in an expression have side-effects, especially when they are > impacted by the evaluation order. That means that the average innocent > future maintainer could break the code by manually rearranging the > expression in a way that's perfectly valid mathematically, e.g. as a > readability fix. > > >> even in the >> presence of (function and variable) type inference, but allow some >> leeway for explicitly declared C functions. > > I'm actually leaning towards not guaranteeing the order of execution if C > doesn't do it either. If this is really required, it's easy to work around > for users, but it's severely hard to fix for Cython in all cases, and the > gain is truly small. After all, we'd only make it easier for users to write > bad code. Yep. Lets keep the code in for the above case. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout wrote: > On 3/16/11 11:05 PM, Robert Bradshaw wrote: >> >> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel >> wrote: >>> >>> I'm actually leaning towards not guaranteeing the order of execution if C >>> doesn't do it either. If this is really required, it's easy to work >>> around >>> for users, but it's severely hard to fix for Cython in all cases, and the >>> gain is truly small. After all, we'd only make it easier for users to >>> write >>> bad code. >> >> Yep. Lets keep the code in for the above case. > > > Is there a huge big warning in the docs? Maybe on this page would be a good > place: http://docs.cython.org/src/userguide/limitations.html This doesn't affect Python functions at all, so I'm not sure it belongs on that page. I agree that it should be mentioned that c(p)def functions have C calling semantics, *including* an unspecified order or argument evaluation. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Thu, Mar 17, 2011 at 1:48 PM, Vitja Makarov wrote: > 2011/3/17 Robert Bradshaw : >> On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout >> wrote: >>> On 3/16/11 11:05 PM, Robert Bradshaw wrote: >>>> >>>> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel >>>> wrote: >>>>> >>>>> I'm actually leaning towards not guaranteeing the order of execution if C >>>>> doesn't do it either. If this is really required, it's easy to work >>>>> around >>>>> for users, but it's severely hard to fix for Cython in all cases, and the >>>>> gain is truly small. After all, we'd only make it easier for users to >>>>> write >>>>> bad code. >>>> >>>> Yep. Lets keep the code in for the above case. >>> >>> >>> Is there a huge big warning in the docs? Maybe on this page would be a good >>> place: http://docs.cython.org/src/userguide/limitations.html >> >> This doesn't affect Python functions at all, so I'm not sure it >> belongs on that page. I agree that it should be mentioned that c(p)def >> functions have C calling semantics, *including* an unspecified order >> or argument evaluation. >> > > As you noticed above, how should be handled def function inlining? > > I guess there are restrictions on def function argument type, so > side-effect isn't issue here. Yep, or at least not near as much of an issue. (I think the C compiler can optimize away an, e.g, extra copy of, e.g, int, double and PyObject* temps in most cases.) - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Thu, Mar 17, 2011 at 2:25 PM, Stefan Behnel wrote: > Robert Bradshaw, 17.03.2011 05:05: >> >> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: >>> >>> I'm actually leaning towards not guaranteeing the order of execution if C >>> doesn't do it either. If this is really required, it's easy to work >>> around >>> for users, but it's severely hard to fix for Cython in all cases, and the >>> gain is truly small. After all, we'd only make it easier for users to >>> write >>> bad code. >> >> Yep. Lets keep the code in for the above case. > > Erm, the current status is that we try to guarantee the order by pushing > everything into temps, thus breaking Sage. What code exactly did you intend > to keep here? I was thinking about guarding it with an if False (or flag in Options.py). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Message system refactoring
On Mon, Mar 21, 2011 at 3:51 AM, Dag Sverre Seljebotn wrote: > On 03/21/2011 11:45 AM, Vitja Makarov wrote: >> >> Now error/warning messages are stored in global variables at >> Cython.Compiler.Errors >> >> I think it's much better to move error handling into some object, >> Main.Context for example. >> >> Some benefits: >> - reduce use of global variables >> - allow more then one cython compiler instance at the time >> - make it much easy to implement -Werror >> - cython directives can affect message system (fast_fail, werror) >> - messages could be easily sorted > > +1. I assume the reason this is not done is simply because it would be a lot > of work and the payback is less than spending time on other stuff. +1. I'm fine with the change, it's just never been as much of a pain point as many other things. > By attaching the error context to "env" and "code" one can avoid a lot of > signature changes. I think transforms should take the context in their > constructor. > > Dag Sverre > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: > Hi, > > there seems to be quite some interest in a project to get parts of CPython > and specifically its stdlib rewritten in Cython. I've copied the latest > python-dev mail below. The relevant part of the thread is here: > > http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 Interesting. > In short, we have strong supporters, but Guido has understandable doubts > against a new (and quite large) dependency and potential semantic > deviations. Reading the list, I think others on the list overestimate the semantic differences. Mostly we're talking about things like is vs. equality for floating point numbers and tracebacks (at least for un-annotated code). It's a valid point that Cython is still under such active development. > But there seem to be cases where slight changes would be > acceptable that Cython compiled modules might introduce, such as emitting > different exception messages, changing Python classes into extension > classes, or even preventing monkey patching in modules that are backed by C > modules anyway. > > It would be helpful to get support from the side of external distributors > that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If > they agreed to test the Cython generated stdlib modules in their > distributions, we could get user feedback that would allow python-dev to > take a well founded decision. > > Do we have any volunteers for trying this out? Both on the side of > distributors and implementors? I think Sage might be willing to give it a try. I'll ask tomorrow as part of a talk I'm giving. Note that due to the way Python's import mechanism works, it would be easy (as a first pass) to make a "cythonize this Python install" which would just compile (a subset of) the .py files and drop .so files next to them. This would require no messing with the Python build system or distribution, easy to test and benchmark, and be easy to clean up. > At the current state of affairs, the implementation could still be financed > by a Python backed GSoC project, although it would be cool if more users > could just step up and simply try to compile and optimise stdlib modules > (preferably without major changes to the code). It's certainly a great way > to show off your Cython skills :). I gave it a try with difflib and it > turned out to be quite easy. > > http://blog.behnel.de/index.php?p=155 > > Reimplementing existing C modules in Cython might, however, be more > interesting for python-dev, but also be a larger undertaking. So a GSoC > might be worth running on that. I think that's a great idea. Would you be willing to mentor such a project. > Note that the latest Cython release does not have generator support yet, and > Vitja's branch on github is not very stable. We will try to get it up to > speed and merged during the workshop next week, at which point it will make > more sense to get this project started than right now. > > Stefan > > > Guido van Rossum, 22.03.2011 00:04: >> >> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. Löwis" wrote: >>> >>> Am 21.03.2011 11:58, schrieb Stefan Behnel: Guido van Rossum, 21.03.2011 03:46: > > Thanks for the clarifications. I now have a much better understanding > of what Cython is. But I'm not sold. For one, your attitude about > strict language compatibility worries me when it comes to the stdlib. Not sure what you mean exactly. Given our large user base, we do worry a lot about things like backwards compatibility, for example. If you are referring to compatibility with Python, I don't think anyone in the project really targets Cython as a a drop-in replacement for a Python runtime. We aim to compile Python code, yes, and there's a hand-wavy idea in the back of our head that we may want a plain Python compatibility mode at some point that will disable several important optimisations. >>> >>> I think that's the attitude Guido worries about: if you don't have the >>> desire to provide 100% Python compatibility under all circumstances >>> (i.e. including if someone passes parameters of "incorrect" types), >>> then there is very little chance that we would replace a Python module >>> with a Cython-compiled one. >>> >>> The only exception would be cases where the Python semantics is murky >>> (e.g. where Jython or so actually behaves differently for the same >>> Python code, and still claims language conformance). E.g. the exact >>> message on a TypeError might change when compiling with Cython, >>> but the cases in which you get a TypeError must not change. >> >> One other significant use case is the situation where we have an >> optional replacement module written in C (e.g. heapqmodule.c vs. >> heapq.py). There are usually many semantic differences between the C >> and pure-python module that we don't care about (e.g. monkeypatching >> won't work). >> >> The size of Cython as a de
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Tue, Mar 22, 2011 at 4:09 PM, Dan Stromberg wrote: > > I think it's a good idea, but I think it'd be better to use pure mode to get > code that runs either way, or some sort of preprocessor (I've used m4 with > good luck for this, though it doesn't syntax highlight nicely) to > automatically derive pure python and cython from the same source file. It doesn't hurt to explore the potential before coming up with the actual solution. Ideally, the .py files would not have to be modified at all. > For me at least, the branch of Cython that supports generators has worked > flawlessly - except for one buglet that prevented use on 2.5.x. It was > quickly fixed when reported though. > > Cython probably should get out of the tiny-version-number phase first > though. I often feel that opensource projects use tiny version numbers for > years as a sort of cop out. well after people have begun relying on the code > for production use. We have a clear 1.0 goal, being able to compile the full Python language. We're not there yet, but very close. It may make sense at that point to also clean up any cruft we don't want to continue supporting forever. I agree, until that point, there's no way we would be a Python development dependency. - Robert > On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >> >> Hi, >> >> there seems to be quite some interest in a project to get parts of CPython >> and specifically its stdlib rewritten in Cython. I've copied the latest >> python-dev mail below. The relevant part of the thread is here: >> >> http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 >> >> In short, we have strong supporters, but Guido has understandable doubts >> against a new (and quite large) dependency and potential semantic >> deviations. But there seem to be cases where slight changes would be >> acceptable that Cython compiled modules might introduce, such as emitting >> different exception messages, changing Python classes into extension >> classes, or even preventing monkey patching in modules that are backed by C >> modules anyway. >> >> It would be helpful to get support from the side of external distributors >> that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If >> they agreed to test the Cython generated stdlib modules in their >> distributions, we could get user feedback that would allow python-dev to >> take a well founded decision. >> >> Do we have any volunteers for trying this out? Both on the side of >> distributors and implementors? >> >> At the current state of affairs, the implementation could still be >> financed by a Python backed GSoC project, although it would be cool if more >> users could just step up and simply try to compile and optimise stdlib >> modules (preferably without major changes to the code). It's certainly a >> great way to show off your Cython skills :). I gave it a try with difflib >> and it turned out to be quite easy. >> >> http://blog.behnel.de/index.php?p=155 >> >> Reimplementing existing C modules in Cython might, however, be more >> interesting for python-dev, but also be a larger undertaking. So a GSoC >> might be worth running on that. >> >> Note that the latest Cython release does not have generator support yet, >> and Vitja's branch on github is not very stable. We will try to get it up to >> speed and merged during the workshop next week, at which point it will make >> more sense to get this project started than right now. >> >> Stefan >> >> >> Guido van Rossum, 22.03.2011 00:04: >>> >>> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. Löwis" wrote: Am 21.03.2011 11:58, schrieb Stefan Behnel: > > Guido van Rossum, 21.03.2011 03:46: >> >> Thanks for the clarifications. I now have a much better understanding >> of what Cython is. But I'm not sold. For one, your attitude about >> strict language compatibility worries me when it comes to the stdlib. > > Not sure what you mean exactly. Given our large user base, we do worry > a > lot about things like backwards compatibility, for example. > > If you are referring to compatibility with Python, I don't think anyone > in the project really targets Cython as a a drop-in replacement for a > Python runtime. We aim to compile Python code, yes, and there's a > hand-wavy idea in the back of our head that we may want a plain Python > compatibility mode at some point that will disable several important > optimisations. I think that's the attitude Guido worries about: if you don't have the desire to provide 100% Python compatibility under all circumstances (i.e. including if someone passes parameters of "incorrect" types), then there is very little chance that we would replace a Python module with a Cython-compiled one. The only exception would be cases where the Python semantics is murky (e.g. where Jython or so actually behaves differently for the same Python cod
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Thu, Mar 24, 2011 at 10:42 AM, Stefan Behnel wrote: > Stefan Behnel, 22.03.2011 08:59: >> >> Robert Bradshaw, 22.03.2011 08:14: >>> >>> On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >>>> >>>> Reimplementing existing C modules in Cython might, however, be more >>>> interesting for python-dev, but also be a larger undertaking. So a GSoC >>>> might be worth running on that. >>> >>> I think that's a great idea. Would you be willing to mentor such a >>> project. >> >> As usual, I'm not sure I'll have the time, but if no-one else steps up, >> I'd >> consider it. > > Should we sign up with the PSF GSoC umbrella to officially propose this > project? > > https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUUVGWmc6MQ > > Do we have any other topics for the GSoC this year? We'd be just in time if > we discuss this at the workshop, although somewhat late already. The student > application deadline is from March 28 to April 8. > > http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline There's always our enhancements page, but some of those are fairly advanced. I think this'd be a really good GSoC project, very modular, and doesn't require getting to know the internals of the compiler to start making a serious contribution (at least not right away, I'm sure bugs and optimizations will be submitted). I'm willing to (co?)-mentor. Stefan? Craig? With the three of us we could easily mentor at least one student, and perhaps two (if another really solid proposal/student comes up). Anyone else willing to mentor? I haven't pushed on GSoC much this year yet because no one's stepped up to mentor, but there's still ample time on our side. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Wed, Mar 23, 2011 at 12:45 AM, Stefan Behnel wrote: > Craig Citro, 23.03.2011 08:11: >>> >>> We have a clear 1.0 goal, being able to compile the full Python >>> language. We're not there yet, but very close. It may make sense at >>> that point to also clean up any cruft we don't want to continue >>> supporting forever. I agree, until that point, there's no way we would >>> be a Python development dependency. >> >> Are we aiming for 100% compatibility, or 99.9%? For instance, I seem to >> recall a few dark corners we aren't planning on covering I started a list at http://wiki.cython.org/Unsupported . I'd say we can be as compatible as Jython/IronPython is, and more than CPython is between minor versions. I would be happy with a short, well-justified list of differences. This will be clearer once the community (which we're a part of) defines what Python vs. implementation details means. > I think it's more like 98%. CPython, PyPy and friends consider the more > obscure features like frame access a core Python language feature, but I > don't see an interest in making that work in Cython. If someone wants to > write the code, fine, as long as it's switched off outside of a future > "strict compatibility mode". I've never seen this used in any code that I > would have wanted to compile with Cython, and even in Python code it's only > really used for more or less "clever hacks". > > >> maybe some details of traceback construction? > > What I would like to see here is simply a fallback of the frames that we > create for tracebacks to look for a .py file (or even .pyx/.pxi file) next > to the compiled .so file, so that they can print the source line for the > executed line in the .so. > > Apart from that, it boils down to the issue of frame construction. See > above. Keep in mind that CPython's own C code currently gives you no frames > at all, so Cython modules already have a serious advantage over what's there > today. > > >> (I want to say multiple inheritance, too, >> but I think that's only an issue for cdef classes, right?) > > Yes. That's how things work in CPython at the C level, and I'm fine with > that. We have multiple inheritance for normal classes, so that's covered. Any auto-cdeffing of classes would have to be done carefully. >> I think it would >> be good to have this written down -- in particular, it seems like there's >> some momentum right now for clearly delineating "Python language >> semantics" >> vs. "CPython implementation detail" in the python-devel community, so it >> might be a particularly good time to raise these questions. > > +1, a "where do we stand?" topic is planned for the workshop anyway. +1 - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Thu, Mar 24, 2011 at 12:58 PM, Stefan Behnel wrote: > Robert Bradshaw, 24.03.2011 20:18: >> >> On Thu, Mar 24, 2011 at 10:42 AM, Stefan Behnel wrote: >>> >>> Stefan Behnel, 22.03.2011 08:59: >>>> >>>> Robert Bradshaw, 22.03.2011 08:14: >>>>> >>>>> On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >>>>>> >>>>>> Reimplementing existing C modules in Cython might, however, be more >>>>>> interesting for python-dev, but also be a larger undertaking. So a >>>>>> GSoC >>>>>> might be worth running on that. >>>>> >>>>> I think that's a great idea. Would you be willing to mentor such a >>>>> project. >>>> >>>> As usual, I'm not sure I'll have the time, but if no-one else steps up, >>>> I'd >>>> consider it. >>> >>> Should we sign up with the PSF GSoC umbrella to officially propose this >>> project? >>> >>> >>> https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUUVGWmc6MQ >>> >>> Do we have any other topics for the GSoC this year? We'd be just in time >>> if >>> we discuss this at the workshop, although somewhat late already. The >>> student >>> application deadline is from March 28 to April 8. >>> >>> >>> http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline >> >> There's always our enhancements page, but some of those are fairly >> advanced. I think this'd be a really good GSoC project, very modular, >> and doesn't require getting to know the internals of the compiler to >> start making a serious contribution (at least not right away, I'm sure >> bugs and optimizations will be submitted). > > Well, almost certainly bugs will be found, at least, and if we get a fix as > well, I'd be the more happy. :) > > >> I'm willing to (co?)-mentor. Stefan? Craig? With the three of us we >> could easily mentor at least one student, and perhaps two (if another >> really solid proposal/student comes up). > > +1, splitting this will make it easier for everyone. And there will be > support from the lists anyway. > > >> Anyone else willing to >> mentor? I haven't pushed on GSoC much this year yet because no one's >> stepped up to mentor, but there's still ample time on our side. > > Ok, want to sign up the Cython project then? Well, I'm talking about under the Python foundation umbrella. > What became of the Sage proposal, BTW? Will they give a compiled stdlib a > try? I completely forgot to ask (too many other things on my mind). We have a fairly extensive test suite, so I think the thing to do would be to try it out and see if anything breaks (which will be a good confidence builder for both Cython and Sage). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Bug in cython-mode.el under GNU Emacs 23.2.1
On Wed, Mar 23, 2011 at 3:48 PM, Lisandro Dalcin wrote: > On 22 March 2011 02:21, Vitja Makarov wrote: >> 2011/3/22 Rafe Kettler : >>> I've just tried out cython-mode.el and I was a bit disappointed when it >>> didn't work right off the bat. >>> >>> I added the following to my .emacs: >>> >>> (setq load-path (cons "~/lib/emacs-plugins" load-path)) >>> (require 'cython-mode) >>> >>> this initially gave me the following error when I restarted emacs: >>> >>> Warning (initialization): An error occurred while loading >>> `/home/rafe/.emacs': >>> >>> File error: Cannot open load file, python-mode >>> >>> To ensure normal operation, you should investigate and remove the >>> cause of the error in your initialization file. Start Emacs with >>> the `--debug-init' option to view a complete error backtrace. >>> >>> So, seeing that python-mode wasn't around, I changed line 3 of >>> cython-mode.el from (require 'python-mode) to (require 'python) and >>> everything works now. >>> >>> I got this mode from Github this morning and it is the latest version (you >>> can see it at >>> https://github.com/cython/cython/blob/master/Tools/cython-mode.el). >>> >>> I'm not an emacs lisp expert, so I'm not sure if there's something I'm >>> missing here or if I somehow loaded the package correctly. Anyway, I thought >>> you guys should know that I had a problem with the file but managed to get a >>> fix. If my fix needs to be applied, I'll patch it on Github. >>> >>> This is GNU Emacs 23.2.1 x64 on Fedora 14, btw. >>> >>> Rafe >>> >> >> Emacs23 have native support for python, you can install python-mode package: >> >> $ sudo apt-get install python-mode >> >> Instead of requiring python it's better to try python-mode first then python: >> >> (when (not (require 'python-mode nil t)) >> (require 'python)) >> > > Should we push this fix? Makes sense to me. Rafe, can you push your changes and do a pull request? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Fri, Mar 25, 2011 at 6:42 AM, Stefan Behnel wrote: > Sturla Molden, 25.03.2011 14:03: >> >> Den 24.03.2011 20:38, skrev Robert Bradshaw: >>> >>> I started a list at http://wiki.cython.org/Unsupported . I'd say we >>> can be as compatible as Jython/IronPython is, and more than CPython is >>> between minor versions. I would be happy with a short, well-justified >>> list of differences. This will be clearer once the community (which >>> we're a part of) defines what Python vs. implementation details means. >> >> Looking at Guido's comment, Cython must be able to compile all valid >> Python if this will have any chance of success. Good thing that's our goal (pending an actual definition of "all valid Python.") > I think there are two levels of required compatibility, the lower one of > which applies to a reimplementation of C modules in Cythen. That's the > reason why I see that as the more worthwhile goal for now. > > >> Is the plan to include Cython in the standard library? I don't think a >> large external dependency like Cython will be accepted unless it's a part >> of the CPython distribution. > > It was the plan a while ago, but the problem is that Cython's evolution (and > stability) isn't very much in line with that of CPython. I'm not as > pessimistic as you seem to be, though. From the POV of CPython, Cython is > basically just a build tool. We could try to come up with a release series > that we consider stable enough to base CPython on it. If we agree to > maintain that for the whole lifetime of the corresponding CPython > release(s), it wouldn't necessarily have to be part of CPython itself > (although it could...) As a first step, CPython would just ship the generated C, only requiring Cython as a development dependency, not a build dependency. This would also be safer from the stability POV. >> Why stop with the standard library? Why not implement the whole CPython >> interpreter in Cython? > > No-one said we'd stop there. ;) Of course the CPython interpreter is a large, fairly-optimized C codebase already, so the pitfalls of just re-writing it for the sake of re-writing it probably outweigh the gains. > However, going down that route will quickly turn into a problem of > boot-strapping. How would you run Cython without a Python interpreter? Would > you need an old interpreter installed in order to run Cython to build a new > one? Currently, you only need a C compiler and cmmi tools. I could easily > understand anyone who'd reject entering into such a recursive dependency. It would be as easy as bootstrapping gcc... :). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Sun, Mar 27, 2011 at 3:39 PM, Sturla Molden wrote: > Den 25.03.2011 19:03, skrev Robert Bradshaw: >> >>>> Looking at Guido's comment, Cython must be able to compile all valid >>>> Python if this will have any chance of success. >> >> Good thing that's our goal (pending an actual definition of "all valid >> Python.") >> > > In lack of a Python language specification it can be hard to tell > implementation details from syntax. It sounded though as if Guido was > worried about Cython's compatibility with Python, and maybe the Cython dev > team's attitude to Python compatibility. We are very concerned about Python compatibility. > Also don't think Cython's main strength in this context was properly > clarified in the debate. It is easy to over-focus on "speed", when it's > really a matter of "writing Python C extensions easily" -- i.e. without > knowing (a lot) about Python's C API, not having to worry about reference > counting, and the possibility of using Python code as prototype. Cython is, > without comparison, the easiest way of writing C extensions for Python. > FWIW, it's easier to use Cython than ctypes. Using Cython instead of the C > API will also avoid many programming errors, because a compiler does fewer > mistakes than a human. Those aspects are important to communicate, not just > "Cython can be as fast as C++". That is a good point. On the other hand, I don't see re-implementing working C modules written, though probably valuable from a maintenance point of view, as compelling of a use case. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cython crash: default parameters in methods of template cppclass
On Tue, Mar 22, 2011 at 7:21 AM, Simon Anders wrote: > Hi, > > I found a little bug in Cython 0.14.1. > > The following causes the Cython compiler to throw an exception: > > ---8<--- > cdef extern from "foo.h": > cdef cppclass foo[ T ]: > bar( int b = 0 ) > > cdef foo[ int ] a > a.bar( 1 ) > ---8<--- > > The exception is "AttributeError: 'CFuncType' object has no attribute > 'op_arg_struct'", thrown in line 3044, in generate_result_code. Full stack > trace attached. > > The file compiles without crash if one either > - removes the last line, i.e. the call to the class's method, or > - removes the template argument, i.e., the "[ T ]" in line 2 and > the "[ int ]" in line 5. > > Hence, the issue only appears when calling methods of _templated_ classes, > which have a _default_ value for their argument. Default arguments in C++ are completely different from default arguments in Cython, this is our bug for not catching that. You have to declare this as an overloaded method cdef extern from "foo.h": cdef cppclass foo[ T ]: bar() bar(int) Btw, unless it returns an object, you probably want to declare the return type of your methods. Perhaps we should make this a warning for extern declarations? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CEP: prange for parallel loops
On Mon, Apr 4, 2011 at 6:04 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 04.04.2011 13:53: >> >> On 04/04/2011 01:23 PM, Stefan Behnel wrote: >>> >>> Dag Sverre Seljebotn, 04.04.2011 12:17: CEP up at http://wiki.cython.org/enhancements/prange >>> >>> """ >>> Variable handling >>> >>> Rather than explicit declaration of shared/private variables we rely on >>> conventions: >>> >>> * Thread-shared: Variables that are only read and not written in the loop >>> body are shared across threads. Variables that are only used in the else >>> block are considered shared as well. >>> >>> * Thread-private: Variables that are assigned to in the loop body are >>> thread-private. Obviously, the iteration counter is thread-private as >>> well. >>> >>> * Reduction: Variables that only used on the LHS of an inplace operator, >>> such as s above, are marked as targets for reduction. If the variable is >>> also used in other ways (LHS of assignment or in an expression) it does >>> instead turn into a thread-private variable. Note: This means that if >>> one, e.g., inserts printf(... s) above, s is turned into a thread-local >>> variable. OTOH, there is simply no way to correctly emulate the effect >>> printf(... s) would have in a sequential loop, so such code must be >>> discouraged anyway. >>> """ >>> >>> What about simply (ab-)using Python semantics and creating a new inner >>> scope for the prange loop body? That would basically make the loop behave >>> like a closure function, but with the looping header at the 'right' place >>> rather than after the closure. >> >> I'm not quite sure what the concrete changes to the CEP this would lead to >> (assuming you mean this as a proposal for alternative semantics, and not >> an >> implementation detail). > > What I would like to avoid is having to tell users "and now for something > completely different". It looks like a loop, but then there's a whole page > of new semantics for it. And this also cannot be used in plain Python code > due to the differing scoping behaviour. The same could be said of OpenMP--it looks exactly like a loop except for a couple of pragmas. The proposed (as I'm reading the CEP now) semantics of what's shared and first/last private and reduction would give it the semantics of a normal, sequential loop (and if your final result changes based on how many threads were involved then you've got incorrect code). Perhaps reading of the reduction variable could be fine (though obviously ill-defined, suitable only for debugging). >> How would we treat reduction variables? They need to be supported, and >> there's nothing in Python semantics to support reduction variables, they >> are a rather special case everywhere. I suppose keeping the reduction >> clause above, or use the "nonlocal" keyword in the loop body... > > That's what I thought, yes. It looks unexpected, sure. That's the clear > advantage of using inner functions, which do not add anything new at all. > But if we want to add something that looks more like a loop, we should at > least make it behave like something that's easy to explain. > > Sorry for not taking the opportunity to articulate my scepticism in the > workshop discussion. Skipping through the CEP now, I think this feature adds > quite some complexity to the language, and I'm not sure it's worth that when > compared to the existing closures. The equivalent closure+decorator syntax > is certainly easier to explain, and could translate into exactly the same > code. But with the clear advantage that the scope of local, nonlocal and > thread-configuring variables is immediately obvious. > > Basically, your example would become > > def f(np.ndarray[double] x, double alpha): > cdef double s = 0 > > with cython.nogil: > @cython.run_parallel_for_loop( range(x.shape[0]) ) > cdef threaded_loop(i): # 'nogil' is inherited > cdef double tmp = alpha * i > nonlocal s > s += x[i] * tmp > s += alpha * (x.shape[0] - 1) > return s > > We likely agree that this is not beautiful. It's also harder to implement > than a "simple" for-in-prange loop. But I find it at least easier to explain > and semantically 'obvious'. And it would allow us to write a pure mode > implementation for this based on the threading module. I'm not opposed to having something like this, it's a whole lot of code and extra refactoring for the basic usecase. I think a nice, clean syntax is worthwhile and requires at lest some level of language support. In some ways it's like buffer support--what goes on under the hood does take some explaining, but most of the time it works as expected (i.e. as if you hadn't declared the type), just faster. The inner workings of prange may be a bit magical, but the intent is not, and the latter is what users care about. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-deve
Re: [Cython] CEP: prange for parallel loops
On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel wrote:
> mark florisson, 04.04.2011 21:26:
>>
>> For clarity, I'll add an example:
>>
>> def f(np.ndarray[double] x, double alpha):
>> cdef double s = 0
>> cdef double tmp = 2
>> cdef double other = 6.6
>>
>> with nogil:
>> for i in prange(x.shape[0]):
>> # reading 'tmp' makes it firstprivate in addition to
>> lastprivate
>> # 'other' is only ever read, so it's shared
>> printf("%lf %lf %lf\n", tmp, s, other)
>
> So, adding a printf() to your code can change the semantics of your
> variables? That sounds like a really bad design to me.
That's what I was thinking. Basically, if you do an inlace operation,
then it's a reduction variable, no matter what else you do to it
(including possibly a direct assignment, though we could make that a
compile-time error).
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CEP: prange for parallel loops
On Tue, Apr 5, 2011 at 4:52 AM, mark florisson
wrote:
> On 5 April 2011 12:51, Stefan Behnel wrote:
>> mark florisson, 04.04.2011 21:26:
>>>
>>> For clarity, I'll add an example:
>>>
>>> def f(np.ndarray[double] x, double alpha):
>>> cdef double s = 0
>>> cdef double tmp = 2
>>> cdef double other = 6.6
>>>
>>> with nogil:
>>> for i in prange(x.shape[0]):
>>> # reading 'tmp' makes it firstprivate in addition to
>>> lastprivate
>>> # 'other' is only ever read, so it's shared
>>> printf("%lf %lf %lf\n", tmp, s, other)
>>
>> So, adding a printf() to your code can change the semantics of your
>> variables? That sounds like a really bad design to me.
>
> I agree, I think we should refrain from the firstprivate() entirely,
> as it wouldn't have the same semantics as serial execution (as 'tmp'
> would have the original value with parallel execution and the value
> from previous iterations with serial execution). So basically we
> should allow reading of private variables only after they are assigned
> to in the loop body.
Unless I'm miss-understanding the meaning of firstprivate (it's
initialized per-thread, not per-iteration), for single-threaded
execution, it would have exactly the same semantics as serial
execution. As I mentioned before, if your code functions differently
for single or multiple threads, then it's incorrect. I think it's
natural that a parallel loop would behave like
tmp = global_value
if fork():
# do first half of the loop, with tmp starting as global_value
else:
# do last half of the loop, with tmp starting as global_value
# reduction magic
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CEP: prange for parallel loops
On Tue, Apr 5, 2011 at 5:55 AM, Pauli Virtanen wrote: > Mon, 04 Apr 2011 21:26:34 +0200, mark florisson wrote: > [clip] >> For clarity, I'll add an example: > [clip] > > How about making all the special declarations explicit? The automatic > inference of variables has a problem in that a small change in a part of > the code can have somewhat unintuitive non-local effects, as the private/ > shared/reduction status of the variable changes in the whole function > scope (if Python scoping is retained). > > Like so with explicit declarations: That's an interesting idea. It's a bit odd specifying the scope as part of the type, but may work. However, I'm still not convinced that we can't safely infer this information. > def f(np.ndarray[double] x, double alpha): > cdef double alpha = 6.6 > cdef char *ptr = something() > > # Parallel variables are declared beforehand; > # the exact syntax could also be something else > cdef cython.parallel.private[int] tmp = 2, tmp2 > cdef cython.parallel.reduction[int] s = 0 > > # Act like ordinary cdef outside prange(); in the prange they are > # firstprivate if initialized or written to outside the loop anywhere > # in the scope. Or, they could be firstprivate always, if this > # has a negligible performance impact. > tmp = 3 > > with nogil: > s = 9 > > for i in prange(x.shape[0]): > if cython.parallel.first_iteration(i): > # whatever initialization; Cython is in principle allowed > # to move this outside the loop, at least if it is > # the first thing here > pass > > # tmp2 is not firstprivate, as it's not written to outside > # the loop body; also, it's also not lastprivate as it's not > # read outside the loop > tmp2 = 99 > > # Increment a private variable > tmp += 2*tmp > > # Add stuff to reduction > s += alpha*i > > # The following raise a compilation error -- the reduction > # variable cannot be assigned to, and can be only operated on > # with only a single reduction operation inside prange > s *= 9 > s = 8 > > # It can be read, however, provided openmp supports this > tmp = s > > # Assignment to non-private variables causes a compile-time > # error; this avoids common mistakes, such as forgetting to > # declare the reduction variable. > alpha += 42 > alpha123 = 9 > ptr = 94 > > # These, however, need to be allowed: > # the users are on their own to make sure they don't clobber > # non-local variables > x[i] = 123 > (ptr + i)[0] = 123 > some_routine(x, ptr, i) > else: > # private variables are lastprivate if read outside the loop > foo = tmp > > # The else: block can be added, but actually has no effect > # as it is always executed --- the code here could as well > # be written after the for loop > foo = tmp # <- same result > > with nogil: > # Suppose Cython allowed cdef inside blocks with usual scoping > # rules > cdef cython.parallel.reduction[double] r = 0 > > # the same variables can be used again in a second parallel loop > for i in prange(x.shape[0]): > r += 1.5 > s -= i > tmp = 9 > > # also the iteration variable is available after the loop > count = i > > # As per usual Cython scoping rules > return r, s > > What did I miss here? As far as I see, the above would have the same > semantics and scoping as a single-threaded Python implementation. One thing is that it's forcing the scope of the variable to be consistant throughout the entire function body, so, for example, a reduction variable in one loop could not be used as a shared in another (without having to declare a new variable), which is a different form of non-locality. > The only change required to make things parallel is replacing range() by > prange() and adding the variable declarations. > > -- > Pauli Virtanen > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CEP: prange for parallel loops
On Tue, Apr 5, 2011 at 8:02 AM, Dag Sverre Seljebotn
wrote:
> On 04/05/2011 04:58 PM, Dag Sverre Seljebotn wrote:
>>
>> On 04/05/2011 04:53 PM, Robert Bradshaw wrote:
>>>
>>> On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel
>>> wrote:
>>>>
>>>> mark florisson, 04.04.2011 21:26:
>>>>>
>>>>> For clarity, I'll add an example:
>>>>>
>>>>> def f(np.ndarray[double] x, double alpha):
>>>>> cdef double s = 0
>>>>> cdef double tmp = 2
>>>>> cdef double other = 6.6
>>>>>
>>>>> with nogil:
>>>>> for i in prange(x.shape[0]):
>>>>> # reading 'tmp' makes it firstprivate in addition to
>>>>> lastprivate
>>>>> # 'other' is only ever read, so it's shared
>>>>> printf("%lf %lf %lf\n", tmp, s, other)
>>>>
>>>> So, adding a printf() to your code can change the semantics of your
>>>> variables? That sounds like a really bad design to me.
>>>
>>> That's what I was thinking. Basically, if you do an inlace operation,
>>> then it's a reduction variable, no matter what else you do to it
>>> (including possibly a direct assignment, though we could make that a
>>> compile-time error).
>>
>> -1, I think that's too obscure. Not being able to use inplace operators
>> for certain variables will be at the very least be nagging.
You could still use inplace operators to your hearts content--just
don't bother using the reduced variable outside the loop. (I guess I'm
assuming reducing a variable has negligible performance overhead,
which it should.) For the rare cases that you want the non-aggregated
private, make an assignment to another variable, or use non-inplace
operations.
Not being able to mix inplace operators might be an annoyance. We
could also allow explicit declarations, as per Pauli's suggestion, but
not require them. Essentially, as long as we have
1) Sequential behavior == one thread scheduled (by semantics)
2) one thread scheduled == multiple threads scheduled (user's
responsibility, as it must be)
then I think we should be fine.
>> I think we need to explicitly declare something. Either a simple
>> prange(..., reduce="s:+"), or all-out declaration of thread-local variables.
>
> Sorry: prange(..., reduce="s"), or perhaps &s or cython.address(s). The + is
> of course still specified in code.
>
> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython paper in Computing in Science & Engineering
On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone wrote: >>> Researchers: Please consider citing this paper if Cython helps your >>> research in non-trivial ways. >> >> Is this the canonical citation reference for Cython now? If so, can this be >> mentioned on the Cython webpage somewhere that is prominent enough to be >> found? > > On a related note, would it be possible to post a preprint somewhere > that isn't behind a paywall? If that's allowed, I would be delighted > to share the preprint with friends to introduce them to Cython. Yes, I think we can post the pre-print, though I'm opposed to making this the "canonical citation" just because of this paywall. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython paper in Computing in Science & Engineering
On Wed, Apr 6, 2011 at 10:54 PM, Dag Sverre Seljebotn wrote: > On 04/07/2011 02:12 AM, Robert Bradshaw wrote: >> >> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone wrote: >>>>> >>>>> Researchers: Please consider citing this paper if Cython helps your >>>>> research in non-trivial ways. >>>> >>>> Is this the canonical citation reference for Cython now? If so, can >>>> this be >>>> mentioned on the Cython webpage somewhere that is prominent enough to be >>>> found? >>> >>> On a related note, would it be possible to post a preprint somewhere >>> that isn't behind a paywall? If that's allowed, I would be delighted >>> to share the preprint with friends to introduce them to Cython. >> >> Yes, I think we can post the pre-print, though I'm opposed to making >> this the "canonical citation" just because of this paywall. > > Is this for ideological or practical reasons? Both. Actually, opposed is probably too strong of a word here. I'm disinclined, but there isn't really a better option. Currently, people usually just cite the website, for whatever that's worth. http://scholar.google.com/scholar?q=cython > This is probably the only paper in a "real" journal for some time, and > citations are going to boost the authors' citation counts. Nobody would > actually look up the citation anyway simply to learn about Cython, they'd > just Google it. So unless we're trying to hide the existence of the paper, I > think we should make it the default citation until there's something better. > > Next time we've got anything to share in a paper, let's do it here: > > http://www.openresearchcomputation.com/ > > Although that wasn't around when we started writing the paper. Or at least look into this more carefully. Some of CiSE's papers are open access, I (naively) thought ours wouldn't be hard to get to either. It is a nice paper though and I think it'll hit a nice audience (who primarily won't even be aware that they're paying through it indirectly through university overhead and monolithic library subscriptions). > Posting the pre-print is a matter of making the necesarry references within > it and formatting it. > > http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615 > <http://www.sherpa.ac.uk/romeo/search.php?jrule=ISSN&search=1521-9615> > > I'll fix it and post a link later today. Thanks. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython paper in Computing in Science & Engineering
On Thu, Apr 7, 2011 at 1:33 AM, Dag Sverre Seljebotn
wrote:
> On 04/07/2011 10:00 AM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 07.04.2011 07:54:
>>>
>>> On 04/07/2011 02:12 AM, Robert Bradshaw wrote:
>>>>
>>>> On Wed, Apr 6, 2011 at 4:40 PM, Zak Stone wrote:
>>>>>>>
>>>>>>> Researchers: Please consider citing this paper if Cython helps your
>>>>>>> research in non-trivial ways.
>>>>>>
>>>>>> Is this the canonical citation reference for Cython now? If so, can
>>>>>> this be
>>>>>> mentioned on the Cython webpage somewhere that is prominent enough to
>>>>>> be
>>>>>> found?
>>>>>
>>>>> On a related note, would it be possible to post a preprint somewhere
>>>>> that isn't behind a paywall? If that's allowed, I would be delighted
>>>>> to share the preprint with friends to introduce them to Cython.
>>>>
>>>> Yes, I think we can post the pre-print, though I'm opposed to making
>>>> this the "canonical citation" just because of this paywall.
>>>
>>> Is this for ideological or practical reasons?
>>
>> Both.
>>
>>
>>> This is probably the only paper in a "real" journal for some time, and
>>> citations are going to boost the authors' citation counts. Nobody would
>>> actually look up the citation anyway simply to learn about Cython, they'd
>>> just Google it.
>>
>> Depends on the reference. If it's just cited as "you know, Cython", people
>> will either look for "Cython" directly and be happy, or they may look up the
>> paper, see that it's paid, and keep searching, either for the paper or for
>> the project. If it's cited as "in that paper, you can read about doing X
>> with Cython", then people will try even harder to get at the paper. In
>> either case, chances are that they need to invest more time because of the
>> reference, compared to a plain link in a footnote. So citing this article is
>> likely to be an inconvenience for interested readers of papers that cite it.
>
> I guess this depends on the paper and reader in question then. Myself I'd
> never bother with the paper but go right to the website. Citing is just
> "paying the authors of the software through improving their citation stats".
> Then again my field is unfortunately very much pyramid-scheme-inflicted.
>
> I definitely think we should encourage giving a footnote as well.
>
> How about just presenting the situation as it is in a "Citing Cython"
> section, and leave the decision up to who's citing Cython? ("If you don't
> like to cite a paywall paper, a website reference is OK. At any rate, please
> link to the website in a footnote the first time you mention Cython.")
Of course eventually it'd be nice if people just wrote "we coded this
up in Cython" and a reference felt as out of place there as if they
had provided a reference for Fortran or C :-). We're a long way from
there though.
> Really, I hate the current situation as much as you do. But I see moving the
> world towards open access as the task of those whose already got a bit up
> the food chain; I'm just at the start of my PhD. (And it should be obvious
> I'm arguing with my own interests in mind here.)
>
> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro wrote: > I've submitted to google the link > is: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1# > It would be really important if you could give me a feedback to my > proposal... > Thank you > Best Regards > Arthur Some quick points: - Python ships with extensive regression tests--use (and possibly augment) those to test your work rather than writing your own. - Three modules for a whole summer seems a bit weak, especially for someone who already knows Cython. Target at least one module/week seems like a good pace; some will be quickies, others might take 40+ hours. And I guarantee you'll get better and faster with practice. - Now that generators are supported, it could also be interesting to look at compiling all the non-C modules and fixing exposed bugs if any, but that might be out of scope. What I'd like to see is an implementation of a single simple but not entirely trivial (e.g. not math) module, passing regression tests with comprable if not better speed than the current C version (though I think it'd probably make sense to start out with the Python version and optimize that). E.g. http://docs.python.org/library/json.html looks like a good candidate. That should only take 8 hours or so, maybe two days at most, given your background. I'm not expecting anything before the application deadline, but if you could whip something like this out in the next week to point to that would help your application out immensely. In fact, one of the Python foundation's requirements is that students submit a patch before being accepted, and this would knock out that requirement and give you a chance to prove yourself. Create an account on https://github.com and commit your code into a new repository there. Hope that helps. - Robert > 2011/4/7 Arthur de Souza Ribeiro >> >> I've wrote a proposal to the project: Reimplement C modules in CPython's >> standard library in Cython. >> I'd be glad if you could take a look a it and give me your feedback. >> the link for the proposal is: http://wiki.cython.org/arthursribeiro >> Thank you. >> Best Regards >> Arthur > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
Looking better. I would add some details about how you plan to compare/profile your new implementations, and also at least add a note that compatibility will be ensured with the Python regression tests. It may make sense to let the exact list of modules be somewhat flexible, for example, based on feedback from the Python and Cython community on what would be the most worthwhile to Cythonize. Maybe the final milestone would be "re-implement several additional modules as chosen by the Python community to provide maximum value" or something like that. - Robert On Thu, Apr 7, 2011 at 11:38 PM, Arthur de Souza Ribeiro wrote: > I've made some changes to my proposal, as you said, I changed the number of > modules I'm going to reimplement, I jumped from three to twelve modules, > what modules are these and when I want to implement is described at: > http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1# > and > http://wiki.cython.org/arthursribeiro > If you could take another look I would appreciate a lot. > Best Regards. > []s > Arthur > > 2011/4/7 Arthur de Souza Ribeiro >> >> >> 2011/4/7 Robert Bradshaw >>> >>> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro >>> wrote: >>> > I've submitted to google the link >>> > >>> > is: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1# >>> > It would be really important if you could give me a feedback to my >>> > proposal... >>> > Thank you >>> > Best Regards >>> > Arthur >>> >>> Some quick points: >>> >>> - Python ships with extensive regression tests--use (and possibly >>> augment) those to test your work rather than writing your own. >> >> Thank you for that information Robert, I didn't realize this. >> >>> >>> - Three modules for a whole summer seems a bit weak, especially for >>> someone who already knows Cython. Target at least one module/week >>> seems like a good pace; some will be quickies, others might take 40+ >>> hours. And I guarantee you'll get better and faster with practice. >> >> I'm going to refactor this Robert, as soon as I remake my project's >> roadmap I'll send to you again. >> >>> >>> - Now that generators are supported, it could also be interesting to >>> look at compiling all the non-C modules and fixing exposed bugs if >>> any, but that might be out of scope. >> >> I will try to take a look at this after implementing some cython code to a >> the module you suggested. >> >>> >>> What I'd like to see is an implementation of a single simple but not >>> entirely trivial (e.g. not math) module, passing regression tests with >>> comprable if not better speed than the current C version (though I >>> think it'd probably make sense to start out with the Python version >>> and optimize that). E.g. http://docs.python.org/library/json.html >>> looks like a good candidate. That should only take 8 hours or so, >>> maybe two days at most, given your background. I'm not expecting >>> anything before the application deadline, but if you could whip >>> something like this out in the next week to point to that would help >>> your application out immensely. In fact, one of the Python >>> foundation's requirements is that students submit a patch before being >>> accepted, and this would knock out that requirement and give you a >>> chance to prove yourself. Create an account on https://github.com and >>> commit your code into a new repository there. >>> >> >> I will start the implementation of json module right now. I created my >> github account and as soon as I have code implemented I will send repository >> link. >> Thanks for all the points you listed, I will work on all of them and send >> an e-mail. >> Best Regards. >> []s >> Arthur >> >>> Hope that helps. >>> >>> - Robert >>> >>> >>> > 2011/4/7 Arthur de Souza Ribeiro >>> >> >>> >> I've wrote a proposal to the project: Reimplement C modules in >>> >> CPython's >>> >> standard library in Cython. >>> >> I'd be glad if you could take a look a it and give me your feedback. >>> >> the link for the proposal is: http://wiki.cython.org/arthursribeiro >>> >> Thank you. >>> >> Best Regards >>> >> Arthur >>> > >>> > ___ >>> > cython-devel mailing list >>> > cython-devel@python.org >>> > http://mail.python.org/mailman/listinfo/cython-devel >>> > >>> > >> > > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
On Fri, Apr 8, 2011 at 10:31 AM, Arthur de Souza Ribeiro wrote: > > > 2011/4/8 Robert Bradshaw >> >> Looking better. I would add some details about how you plan to >> compare/profile your new implementations, and also at least add a note >> that compatibility will be ensured with the Python regression tests. > > I'm going to update this on the wiki page, it's just because the application > period in google-melange ends today and it's not recommended to put details > os how you plan to make the project in there. About the comparison, I wonder > talk to the community a little more about it, because I see in cython's web > page and in the tutorials numbers telling how much more efficient cython is > against python, so I planned to use the same strategy that is used to show > the numbers that are there. And about the tests, I'm studying how can I > check compatibility and do these tasks to put that on wiki too. > >> >> It may make sense to let the exact list of modules be somewhat >> flexible, for example, based on feedback from the Python and Cython >> community on what would be the most worthwhile to Cythonize. Maybe the >> final milestone would be "re-implement several additional modules as >> chosen by the Python community to provide maximum value" or something >> like that. > > You mean just to tell how many modules I'm going to re-implement, but not > telling what modules are these? Re-implementing by community demand? Yes, exactly (for the last milestone, I think it's good to have more direction at the start as well as have something to point to when soliciting feedback. Maybe say "at least three" as it might be some big ones or a handful of little ones. You, I, and everyone else will have a better idea of what'll be most profitable at this point. > Thank you very much again. No problem, thanks for your interest. > Best Regards. > []s > Arthur > >> >> - Robert >> >> On Thu, Apr 7, 2011 at 11:38 PM, Arthur de Souza Ribeiro >> wrote: >> > I've made some changes to my proposal, as you said, I changed the number >> > of >> > modules I'm going to reimplement, I jumped from three to twelve modules, >> > what modules are these and when I want to implement is described at: >> > >> > http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1# >> > and >> > http://wiki.cython.org/arthursribeiro >> > If you could take another look I would appreciate a lot. >> > Best Regards. >> > []s >> > Arthur >> > >> > 2011/4/7 Arthur de Souza Ribeiro >> >> >> >> >> >> 2011/4/7 Robert Bradshaw >> >>> >> >>> On Thu, Apr 7, 2011 at 2:31 PM, Arthur de Souza Ribeiro >> >>> wrote: >> >>> > I've submitted to google the link >> >>> > >> >>> > >> >>> > is: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/arthur_sr/1# >> >>> > It would be really important if you could give me a feedback to my >> >>> > proposal... >> >>> > Thank you >> >>> > Best Regards >> >>> > Arthur >> >>> >> >>> Some quick points: >> >>> >> >>> - Python ships with extensive regression tests--use (and possibly >> >>> augment) those to test your work rather than writing your own. >> >> >> >> Thank you for that information Robert, I didn't realize this. >> >> >> >>> >> >>> - Three modules for a whole summer seems a bit weak, especially for >> >>> someone who already knows Cython. Target at least one module/week >> >>> seems like a good pace; some will be quickies, others might take 40+ >> >>> hours. And I guarantee you'll get better and faster with practice. >> >> >> >> I'm going to refactor this Robert, as soon as I remake my project's >> >> roadmap I'll send to you again. >> >> >> >>> >> >>> - Now that generators are supported, it could also be interesting to >> >>> look at compiling all the non-C modules and fixing exposed bugs if >> >>> any, but that might be out of scope. >> >> >> >> I will try to take a look at this after implementing some cython code >> >> to a >> >> the module you suggested. >> >> >> >>> >> >>
Re: [Cython] cython-docs repository
On Sat, Apr 9, 2011 at 4:49 AM, Jason Grout wrote: > What is the relationship between the cython-docs repository and the docs/ > subdirectory of the cython repository? I see a recent commit [1] that seems > to indicate that cython-docs has been merged into the main cython repository > (+1 from me, for what it's worth). Is my interpretation correct, and is > cython-docs now deprecated? Yep, we did that during the workshop. I thought I had sent out an announcement, but I guess not. > I submitted a pull request for some typos [2], but I'm not sure if I should > have submitted that pull request to the cython-docs repository. Thanks, I'll take a look. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Test runner
On Mon, Apr 11, 2011 at 3:56 AM, mark florisson wrote: > On 11 April 2011 12:53, mark florisson wrote: >> On 11 April 2011 12:45, Stefan Behnel wrote: >>> mark florisson, 11.04.2011 12:26: Can we select tests in the tests directory selectively? I see the -T or --ticket option, but it doens't seem to find the test tagged with # ticket:. I can select unit tests using python runtests.py Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same thing for tests in the tests directory. Running the entire suite takes rather long. >>> >>> You can still select them by name using a regex, e.g. >>> >>> runtests.py 'run\.empty_builtin_constructors' >>> >>> Stefan >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >> >> Great, thanks! I'll update the hackerguide wiki. >> > I see now that it is briefly mentioned there, apologies. I've added a note there about tags as well, and fixed the -T to look at the ticket tag. Note that "mode:run" is the default, so you don't need to explicitly tag mode except for compile/error tests. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cython-docs repository
On Sat, Apr 9, 2011 at 10:13 AM, Jason Grout wrote: > On 4/9/11 12:02 PM, Robert Bradshaw wrote: >> >> Yep, we did that during the workshop. I thought I had sent out an >> announcement, but I guess not. > > Is there a summary anywhere of the exciting things that happened in the > workshop? Not yet, but I'll post as soon as I have a writeup. > For example, it seems that generators are finally in, if I read > the commit logs correctly. Is that true? If so, fantastic! Yep! > Any idea of a timeline for that to make it into an official release? There's still a some fallout and more testing to do, but hopefully it won't be too long before a release. The biggest step back seems to be the disabling of inline generators, now they're full-fledged generators, which is an optimization regression but not a feature regression. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] "Cython's Users Guide"
On Mon, Apr 11, 2011 at 2:35 PM, Francesc Alted wrote: > 2011/4/11 William Stein >> >> Hi, >> >> I'm teaching Cython in my Sage course yet again, and noticed that >> again there are some very confusing aspects of the Cython >> documentation organization, which could probably be improved by a few >> simple changes. >> >> 1. At http://cython.org/ there is a big link in the middle of the >> page labeled "Cython Users Guide" which goes to >> http://docs.cython.org/. However, http://docs.cython.org/ is *not* >> the users guide -- it is "Cython’s Documentation". In fact, the >> Users Guide is Chapter 3 of the documentation. >> >> 2. Looking at http://docs.cython.org, we see that Chapter 2 is >> "Tutorials". But then looking down to Chapter 3 we see that it is >> "Cython Users Guide". Of course, that's what one is after having just >> clicked a link called "Cython Users Guide". So we click on "Cython >> Users Guide" again. >> >> 3. We arrive at a page that again has "Tutorial" as Chapter 2. For >> some reason this makes me feel even more confused. >> >> Recommend changes: >> >> 1. Change the link on the main page from "Cython Users Guide" to >> "Documentation" or put a direct link into the Users Guide, or have >> two links. >> >> 2. At http://docs.cython.org/ rename the "Cython Users Guide" to >> "Users Guide", since it is obviously the Cython Users Guide at this >> point and "Cython documentation" is in the upper left of the page >> everywhere. >> >> 3. Possibly rename the tutorial in chapter 2 of the users guide to >> something like "First Steps" or "Basic Tutorial" or something. Thanks for the suggestions. Done. > Yeah, that's something that we discussed in the past workshop in Munich > (BTW, many thanks for providing the means for making this happen!). The > basic idea is to completely remove the Chapter 3 (Cython Users Guide) by > moving its parts to either Chapter 2 (Tutorials), or either to Chapter 4 > (Reference Guide). During the meeting we agreed that the doc repository > should be moved (and has been moved indeed) into the source repo, so that > modifications that affect to code and docs can be put in the same > commit/branch. Also, the wiki has a lot of information that can be better > consolidated and integrated into the User's Guide. > In fact, I already started some job in this direction and created a couple > of pull requests during the workshop (that they have been already > integrated). I plan to continue this job, but unfortunately I'm pretty busy > lately, so I don't think I can progress a lot in the next weeks, so if > anybody is interested in joining the effort for improving Cython's > documentation, she will be very welcome indeed! +1, and thanks, Fransesc, for the work you've started in moving us in this direction. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
On Tue, Apr 12, 2011 at 11:22 AM, Stefan Behnel wrote: > Arthur de Souza Ribeiro, 12.04.2011 14:59: >> >> Hi Stefan, yes, I'm working on this, in fact I'm trying to recompile json >> module (http://docs.python.org/library/json.html) adding some type >> definitions and cython things o get the code faster. > > Cool. > > >> I'm getting in trouble with some things too, I'm going to enumerate here >> so >> that, you could give me some tips about how to solve them. >> >> 1 - Compile package modules - json module is inside a package (files: >> __init__.py, decoder.py, encoder.py, decoder.py) is there a way to >> generate >> the cython modules just like its get generated by cython? > > The __init__.py doesn't really look performance critical. It's better to > leave that modules in plain Python, that improves readability by reducing > surprises and simplifies reuse by other implementations. > > That being said, you can compile each module separately, just use the > "cython" command line tool for that, or write a little distutils script as > in > > http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils > > Don't worry too much about a build integration for now. > > >> 2 - Because I'm getting in trouble with issue #1, I'm running the tests >> manually, I go to %Python-dir%/Lib/tests/json_tests, get the files >> corresponding to the tests python make and run manually. > > That's fine. > > >> 3 - To get the performance of the module, I'm thinking about to use the >> timeit function in the unit tests for the project. I think a good number >> of >> executions would be made and it would be possible to compare each time. > > That's ok for a start, artificial benchmarks are good to test specific > functionality. However, unit tests tend to be short running with a lot of > overhead, so later on, you will need to use real code to benchmark the > modules. I would expect that there are benchmarks for JSON implementations > around, and you can just generate a large JSON file and run loads and dumps > on it. > > >> 4 - I didn't create the .pxd files, some problems are happening, it tells >> methods are not defined, but, they are defined, I will try to investigate >> this better > > When reporting usage related problems (preferably on the cython-users > mailing list), it's best to present the exact error messages and the > relevant code snippets, so that others can quickly understand what's going > on and/or reproduce the problem. > > >> The code is in this repository: >> https://github.com/arthursribeiro/JSON-module your feedback would be very >> important, so that I could improve my skills to get more and more able to >> work sooner in the project. > > I'd strongly suggest implementing this in pure Python (.py files instead of > .pyx files), with externally provided static types for performance. A single > code base is very advantageous for a large project like CPython, much more > than the ultimate 5% better performance. While this is advantageous for the final product, it may not be the easiest to get up and running with. >> I think some things implemented in this rewriting process are going to be >> useful when doing this with C modules... > > Well, if you can get the existing Python implementation up to mostly > comparable speed as the C implementation, then there is no need to care > about the C module anymore. Even if you can get only 90% of a module to run > at comparable speed, and need to keep 10% in plain C, that's already a huge > improvement in terms of maintainability. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
On Tue, Apr 12, 2011 at 8:33 AM, Sturla Molden wrote: > Den 12.04.2011 14:59, skrev Arthur de Souza Ribeiro: >> >> 1 - Compile package modules - json module is inside a package (files: >> __init__.py, decoder.py, encoder.py, decoder.py) is there a way to generate >> the cython modules just like its get generated by cython? >> > > > I'll propose these 10 guidelines: > > 1. The major concern is to replace the manual use of Python C API with > Cython. We aim to improve correctness and readability, not speed. Speed is a concern, otherwise many of these modules wouldn't have been written in C in the first place (at least, not the ones with a pure Python counterpart). Of course some of them are just wrapping C libraries where speed doesn't matter as much. > 2. Replacing plain C with Cython for readability is less important, > sometimes even discourged. Huh? I'd say this is a big point of the project. Maybe less so than manual dependance on the C API, but certainly not discouraged. > If you do, it's ok to leverage on Python > container types if it makes the code concise and readable, even if it will > sacrifice some speed. That's true. > 3. Use exceptions instead of C style error checks: It's better to ask > forgiveness than permission. Yep, this is natural in Cython. > 4. Use exceptions correctly. All resourse C allocation belongs in __cinit__. > All C resource deallocation belongs in __dealloc__. Remember that exceptions > can cause resource leaks if you don't. Wrap all resource allocation in an > extension type. Never use functions like malloc or fopen directly in your > Cython code, except in a __cinit__ method. This can be useful advice, but is not strictly necessary. Try..finally can fill this need as well. > 5. We should keep as much of the code in Python as we can. Replacing Python > with Cython for speed is less important. Only the parts that will really > benefit from static typing should be changed to Cython. True. Of course, compiling the (unchanged) pure Python files with Cython could also yield interesting results, but that's not part of the project. > 6. Leave the __init__.py file as it is. A Python package is allowed contain > a mix of Python source files and Cython extension libraries. > > 7. Be careful to release the GIL whenever appropriate, and never release it > otherwise. Don't yield the GIL just because you can, it does not come for > free, even with a single thread. > > 8. Use the Python and C standard libraries whenever you can. Don't > re-invent the wheel. Don't use system dependent APIs when the standard > libraries declare a common interface. Callbacks to Python are ok. > > 9. Write code that will work correctly on 32 and 64 bit systems, big- or > little-endian. Know your C: Py_intptr_t can contain a pointer. Py_ssize_t > can represent the largest array size allowed. Py_intptr_t and Py_ssize_t can > have different size. The native array offset can be different from > Py_ssize_t, for which a common example is AMD64. It's rare to have to do pointer arithmetic in Cython, and rarer still to have to store the pointer as an integer. > 10. Don't clutter the namespace, use pxd includes. Short source files are > preferred to long. Simple is better than complex. Keep the source nice and > tidy. Not sure what you mean by "pxd includes," but yes, you should use pxd files and cimport just as you would in Python to keep things manageable and modular. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Test runner
On Wed, Apr 13, 2011 at 4:07 AM, mark florisson
wrote:
> Another, different but related issue: how can we get useful output
> from the test runner? e.g. I'm running my test with a
> '@cython.test_assert_path_exists("...")' and I get this error output:
>
> ==
> ERROR: runTest (__main__.CythonRunTestCase)
> compiling (c) and running parallel
> --
> Traceback (most recent call last):
> File "runtests.py", line 555, in run
> self.runCompileTest()
> File "runtests.py", line 386, in runCompileTest
> self.test_directory, self.expect_errors, self.annotate)
> File "runtests.py", line 532, in compile
> self.assertEquals(None, unexpected_error)
> AssertionError: None != u'9:0: Compiler crash in TreeAssertVisitor'
>
> So I'm seeing a traceback from the test runner (which I'm not really
> interested in :), but the actual traceback is not displayed.
I agree this could be improved, but I'm not sure the best way to do it.
> Can I also specify special link and compiler flags for a certain test,
> like in http://wiki.cython.org/enhancements/distutils_preprocessing ?
> Or do I have to export LDFLAGS and CFLAGS in my environment?
You can't right now, but it would probably be worth adding. I'm not
sure how we would handle missing dependancies in that case though.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] speed.pypy.org
On Fri, Apr 15, 2011 at 1:20 PM, Stefan Behnel wrote: > Stefan Behnel, 11.04.2011 15:08: >> >> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython >> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it >> works out well, we may have it up in a couple of days. > > ... or maybe not. It may take a little longer due to lack of time on his > side. > > >> I would expect that Cython won't be a big winner in this game, given that >> it will only compile plain untyped Python code. It's also going to fail >> entirely in some of the benchmarks. But I think it's worth having it up >> there, simply as a way for us to see where we are performance-wise and to >> get quick (nightly) feed-back about optimisations we try. The benchmark >> suite is also a nice set of real-world Python code that will allow us to >> find compliance issues. > > Ok, here's what I have so far. I fixed a couple of bugs in Cython and got at > least some of the benchmarks running. Note that they are actually simple > ones, only a single module. Basically all complex benchmarks fail due to > known bugs, such as Cython def functions not accepting attribute assignments > (e.g. on wrapping). There's also a problem with code that uses platform > specific names conditionally, such as WindowsError when running on Windows. > Cython complains about non-builtin names here. I'm considering to turn that > into a visible warning instead of an error, so that the name would instead > be looked up dynamically to let the code fail at runtime *iff* it reaches > the name lookup. Given the usefulness of the error, and the (relative) lack of issues with it so far, I'd rather not turn it into only a warning by default (though an option might be nice). Another option would be to whitelist the presumably small, finite set of names that are platform-dependent. > Anyway, here are the numbers. I got them with "auto_cpdef" enabled, although > that doesn't even seem to make that a big difference. The baseline is a > self-compiled Python 2.7.1+ (about a month old). Cool. So basically everything is faster, usually somewhere between a 50-100% improvement. There's lots of room for improvement, though a JIT has a significant advantage that we don't get for untyped code. - Robert > ### ai ### > Min: 0.402407 -> 0.362190: 1.1110x faster > Avg: 0.408784 -> 0.366898: 1.1142x faster > Significant (t=10.017195, a=0.95) > Stddev: 0.00824 -> 0.00442: 1.8668x smaller > > > ### chaos ### (with a bug fixed in the benchmark itself) > Min: 0.393362 -> 0.231932: 1.6960x faster > Avg: 0.401941 -> 0.234089: 1.7170x faster > Significant (t=36.128709, a=0.95) > Stddev: 0.01004 -> 0.00267: 3.7538x smaller > > > ### crypto_pyaes ### > Min: 2.629560 -> 1.276433: 2.0601x faster > Avg: 2.639409 -> 1.277742: 2.0657x faster > Significant (t=368.652396, a=0.95) > Stddev: 0.00812 -> 0.00153: 5.3215x smaller > > > ### fannkuch ### > Min: 1.512630 -> 0.853309: 1.7727x faster > Avg: 1.522860 -> 0.860237: 1.7703x faster > Significant (t=118.573908, a=0.95) > Stddev: 0.00880 -> 0.00887: 1.0073x larger > > > ### float ### > Min: 0.452620 -> 0.343341: 1.3183x faster > Avg: 0.475137 -> 0.349356: 1.3600x faster > Significant (t=9.575876, a=0.95) > Stddev: 0.02838 -> 0.00757: 3.7489x smaller > > > ### go ### > Min: 0.758998 -> 0.491929: 1.5429x faster > Avg: 0.764110 -> 0.496518: 1.5389x faster > Significant (t=90.848407, a=0.95) > Stddev: 0.00400 -> 0.00523: 1.3096x larger > > > ### nbody_modified ### > Min: 0.399168 -> 0.197931: 2.0167x faster > Avg: 0.401379 -> 0.203112: 1.9762x faster > Significant (t=42.377829, a=0.95) > Stddev: 0.00293 -> 0.01004: 3.4337x larger > > > ### raytracesimple ### (module renamed from "raytrace-simple") > Min: 2.016425 -> 1.182970: 1.7045x faster > Avg: 2.030030 -> 1.192164: 1.7028x faster > Significant (t=78.219481, a=0.95) > Stddev: 0.02184 -> 0.00983: 2.2211x smaller > > > ### richards ### > Min: 0.286723 -> 0.162430: 1.7652x faster > Avg: 0.289933 -> 0.165193: 1.7551x faster > Significant (t=52.898468, a=0.95) > Stddev: 0.00392 -> 0.00352: 1.1127x smaller > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] speed.pypy.org
On Sat, Apr 16, 2011 at 1:20 AM, Stefan Behnel wrote: > Robert Bradshaw, 16.04.2011 08:53: >> >> On Fri, Apr 15, 2011 at 1:20 PM, Stefan Behnel wrote: >>> >>> Stefan Behnel, 11.04.2011 15:08: >>>> >>>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get >>>> Cython >>>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it >>>> works out well, we may have it up in a couple of days. >>>> >>>> I would expect that Cython won't be a big winner in this game, given >>>> that >>>> it will only compile plain untyped Python code. It's also going to fail >>>> entirely in some of the benchmarks. But I think it's worth having it up >>>> there, simply as a way for us to see where we are performance-wise and >>>> to >>>> get quick (nightly) feed-back about optimisations we try. The benchmark >>>> suite is also a nice set of real-world Python code that will allow us to >>>> find compliance issues. >>> >>> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got >>> at >>> least some of the benchmarks running. Note that they are actually simple >>> ones, only a single module. Basically all complex benchmarks fail due to >>> known bugs, such as Cython def functions not accepting attribute >>> assignments >>> (e.g. on wrapping). There's also a problem with code that uses platform >>> specific names conditionally, such as WindowsError when running on >>> Windows. >>> Cython complains about non-builtin names here. I'm considering to turn >>> that >>> into a visible warning instead of an error, so that the name would >>> instead >>> be looked up dynamically to let the code fail at runtime *iff* it reaches >>> the name lookup. >> >> Given the usefulness of the error, and the (relative) lack of issues >> with it so far, I'd rather not turn it into only a warning by default >> (though an option might be nice). Another option would be to whitelist >> the presumably small, finite set of names that are platform-dependent. > > I agree, this has caught countless bugs in the past. I think a whitelist > makes sense, but note that this does not obey Python semantics, either. In > Python, any unknown name is just fine as long as it's not being looked up. > Even though the use cases for this are clearly less common than the cases > where it bites users. Well, we certainly want to provide a way for users to disable it, and would trigger it with the -pedantic flag. > I'm currently changing the builtins caching support to simply not cache > unknown names, so that they will be looked up at runtime at the point where > they are used (even though, of cause, they are compile time errors by > default). In combination with a whitelist and with an option to make unknown > builtins a warning instead of an error, this will give us a pretty good > default trade-off between Python semantics, safety and performance, with an > easy option for better Python compatibility. +1 >>> Anyway, here are the numbers. I got them with "auto_cpdef" enabled, >>> although >>> that doesn't even seem to make that a big difference. The baseline is a >>> self-compiled Python 2.7.1+ (about a month old). >> >> Cool. So basically everything is faster, usually somewhere between a >> 50-100% improvement. There's lots of room for improvement, though a >> JIT has a significant advantage that we don't get for untyped code. > > Sure, we won't be as fast as PyPy for plain untyped Python code. But the > benchmark suite gives us a clear target, both in terms of performance and > compatibility. My thoughts as well. With good control flow in place, we can also look into generating multiple branches for code (e.g. the likely, fast one that assumes no overflows, etc. and bailing to the slow, safe one.) - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Code examples missing in Cython User's Guide
On Thu, Apr 14, 2011 at 12:05 PM, Chris Lasher wrote: > Thanks Arthur. I actually found the code examples in a grandparent directory > of the User's Guide documentation source directory. I have submitted a pull > request on GitHub which corrects the User's Guide Tutorial documentation so > that it now includes the code. > > https://github.com/cython/cython/pull/24 > > Could a Cython dev please review and approve this change Done. > and then build and > update the docs on the Cython website for the sake of other Cython newbies? This will be easier to do once we have a separate maintenance branch, as we'd rather the documentation match the current, not next, release. I don't think we (yet) have any new features mentioned in the docs, so I'll go ahead an re-generate them (there's been a lot of good cleanup in there, mostly thanks to Francesc Alted). The lastest docs can be found at https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/. - Robert > On Tue, Apr 12, 2011 at 1:39 PM, Arthur de Souza Ribeiro > wrote: >> >> Hey Chris, the code for primes and fib examples, are in the directory >> 'Demos' of the repository... >> Best Regards. >> []s >> Arthur >> 2011/4/12 Chris Lasher >>> >>> My apologies for cross-posting this from cython-users, but I realized I >>> should have sent this bug report to cython-devel. >>> >>> Several code examples are missing from the User's Guide due to the source >>> code files being moved or deleted. See for example the Tutorial page on the >>> User's Guide http://docs.cython.org/src/userguide/tutorial.html The code for >>> both fib.pyx and primes.pyx (and their setup.py files) is absent from the >>> document. >>> >>> I looked at the ReST files and they are trying to source the files from >>> an "examples" directory. Looking through the git repository, I wasn't able >>> to locate this directory. Has it been moved or deleted? >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >> > > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] gilnanny
On Mon, Apr 18, 2011 at 12:08 PM, mark florisson wrote: > Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get() > for every refnanny inc- and decref, so that it will issue a fatal > error whenever reference counting is done without the gil, to make > sure we never do any illegal things in nogil code blocks. Sounds like a good idea to me. > While I'm at it, I also want to add a pystate.pxd to the cpython includes. Sure, corresponding to pystate.h? Have you looked into how stable this API is (e.g. Py2 vs. Py3)? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] On cdef extern from syntax
On Wed, Apr 20, 2011 at 11:08 AM, Fabrizio Milo aka misto
wrote:
> Hi,
>
> I was wondering if has been ever discussed to implement the cython
> cdef extern from syntax as a with statement:
>
> with cython.header_importer("") as cy:
> cy.ctypedef(" unsigned int uint8 ")
> cy.cfunc( " void * myfunc() )")
>
> or also
>
> with cython.header_importer("") as cy:
> cy("""
> ctypdef ...
>
> """)
>
> Maybe there is already something like that ?
I don't think the with statement really makes sense here, as it is a
list of global declarations, and the with statement is all about
making something local to a block.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Mon, Apr 18, 2011 at 7:51 AM, mark florisson wrote: > On 18 April 2011 16:41, Dag Sverre Seljebotn > wrote: >> Excellent! Sounds great! (as I won't have my laptop for some days I can't >> have a look yet but I will later) >> >> You're right about (the current) buffers and the gil. A testcase explicitly >> for them would be good. >> >> Firstprivate etc: i think it'd be nice myself, but it is probably better to >> take a break from it at this point so that we can think more about that and >> not do anything rash; perhaps open up a specific thread on them and ask for >> more general input. Perhaps you want to take a break or task-switch to >> something else (fused types?) until I can get around to review and merge >> what you have so far? You'll know best what works for you though. If you >> decide to implement explicit threadprivate variables because you've got the >> flow I certainly wom't object myself. >> > Ok, cool, I'll move on :) I already included a test with a prange and > a numpy buffer with indexing. Wow, you're just plowing away at this. Very cool. +1 to disallowing nested prange, that seems to get really messy with little benefit. In terms of the CEP, I'm still unconvinced that firstprivate is not safe to infer, but lets leave the initial values undefined rather than specifying them to be NaNs (we can do that as an implementation if you want), which will give us flexibility to change later once we've had a chance to play around with it. The "cdef threadlocal(int) foo" declaration syntax feels odd to me... We also probably want some way of explicitly marking a variable as shared and still be able to assign to/flush/sync it. Perhaps the parallel context could be used for these declarations, i.e. with parallel(threadlocal=a, shared=(b,c)): ... which would be considered an "expert" usecase. For all the discussion of threadsavailable/threadid, the most common usecase I see is for allocating a large shared buffer and partitioning it. This seems better handled by allocating separate thread-local buffers, no? I still like the context idea, but everything in a parallel block before and after the loop(s) also seems like a natural place to put any setup/teardown code (though the context has the advantage that __exit__ is always called, even if exceptions are raised, which makes cleanup a lot easier to handle). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Thu, Apr 21, 2011 at 1:59 AM, mark florisson
wrote:
> On 21 April 2011 10:37, Robert Bradshaw wrote:
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>> wrote:
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn
>>> wrote:
>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>> have a look yet but I will later)
>>>>
>>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>>> for them would be good.
>>>>
>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>>> take a break from it at this point so that we can think more about that and
>>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>>> more general input. Perhaps you want to take a break or task-switch to
>>>> something else (fused types?) until I can get around to review and merge
>>>> what you have so far? You'll know best what works for you though. If you
>>>> decide to implement explicit threadprivate variables because you've got the
>>>> flow I certainly wom't object myself.
>>>>
>>> Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> Yes, they are currently undefined (and not initialized to NaN etc).
> The thing is that without the control flow analysis (or perhaps not
> until runtime) you won't know whether a variable is initialized at all
> before the parallel section, so making it firstprivate might actually
> copy an undefined value (perhaps with a trap representation!) into the
> thread-private copy, which might invalidate valid code. e.g. consider
>
> x_is_initialized = False
> if condition:
> x = 1
> x_is_initialized = True
>
> for i in prange(10, schedule='static'):
> if x_is_initialized:
> printf("%d\n", x)
> x = i
I'm still failing to see how this is a problem (or anything new, as
opposed to this same example with an ordinary range).
>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>> with parallel(threadlocal=a, shared=(b,c)):
>> ...
>>
>> which would be considered an "expert" usecase.
>
> Indeed, assigning to elements in an array instead doesn't seem very
> convenient :)
>
>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> Currently 'with gil' isn't merged into that branch, and if it will, it
> will be disallowed, as I'm not yet sure how (if at all) it could be
> handled with regard to exceptions. It seems a lot easier to disallow
> it and have the user write a 'with gil' function, from which nothing
> can propagate.
Not being able to propagate exceptions is a pretty strong
constraint--even if the implementation doesn't yet support it, it'd be
nice to have an API that makes it possible as a future feature.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Thu, Apr 21, 2011 at 2:21 AM, mark florisson
wrote:
> On 21 April 2011 10:59, mark florisson wrote:
>> On 21 April 2011 10:37, Robert Bradshaw wrote:
>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>> wrote:
>>>> On 18 April 2011 16:41, Dag Sverre Seljebotn
>>>> wrote:
>>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>>> have a look yet but I will later)
>>>>>
>>>>> You're right about (the current) buffers and the gil. A testcase
>>>>> explicitly
>>>>> for them would be good.
>>>>>
>>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better
>>>>> to
>>>>> take a break from it at this point so that we can think more about that
>>>>> and
>>>>> not do anything rash; perhaps open up a specific thread on them and ask
>>>>> for
>>>>> more general input. Perhaps you want to take a break or task-switch to
>>>>> something else (fused types?) until I can get around to review and merge
>>>>> what you have so far? You'll know best what works for you though. If you
>>>>> decide to implement explicit threadprivate variables because you've got
>>>>> the
>>>>> flow I certainly wom't object myself.
>>>>>
>>>> Ok, cool, I'll move on :) I already included a test with a prange and
>>>> a numpy buffer with indexing.
>>>
>>> Wow, you're just plowing away at this. Very cool.
>>>
>>> +1 to disallowing nested prange, that seems to get really messy with
>>> little benefit.
>>>
>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>> safe to infer, but lets leave the initial values undefined rather than
>>> specifying them to be NaNs (we can do that as an implementation if you
>>> want), which will give us flexibility to change later once we've had a
>>> chance to play around with it.
>>
>> Yes, they are currently undefined (and not initialized to NaN etc).
>> The thing is that without the control flow analysis (or perhaps not
>> until runtime) you won't know whether a variable is initialized at all
>> before the parallel section, so making it firstprivate might actually
>> copy an undefined value (perhaps with a trap representation!) into the
>> thread-private copy, which might invalidate valid code. e.g. consider
>>
>> x_is_initialized = False
>> if condition:
>> x = 1
>> x_is_initialized = True
>>
>> for i in prange(10, schedule='static'):
>> if x_is_initialized:
>> printf("%d\n", x)
>> x = i
>
> Erm, that snippet I posted is invalid in any case, as x will be
> private. So guess initializing things to NaN in such would have to
> occur in the parallel section that should enclose the for. So e.g.
> we'd have to do
>
> #pragma omp parallel private(x)
> {
> x = INT_MAX;
> #pragma omp for lastprivate(i)
> for (...)
> ...
> }
>
> Which would then mean that 'x' cannot be lastprivate anymore :). So
> it's either "uninitialized and undefined" or "firstprivate". I
> personally prefer the former for the implicit route.
A variable can't be both first and last private? In any case, as long
as we don't promise anything about them now, we can decide later.
> I do like the threadlocal=a stuff to parallel, it's basically what I
> proposed a while back except that you don't make them strings, but
> better because most of your variables can be inferred, so the
> messiness is gone.
Yep.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Thu, Apr 21, 2011 at 11:13 AM, Dag Sverre Seljebotn wrote: > On 04/21/2011 10:37 AM, Robert Bradshaw wrote: >> >> In terms of the CEP, I'm still unconvinced that firstprivate is not >> safe to infer, but lets leave the initial values undefined rather than >> specifying them to be NaNs (we can do that as an implementation if you >> want), which will give us flexibility to change later once we've had a >> chance to play around with it. > > I don't see any technical issues with inferring firstprivate, the question > is whether we want to. I suggest not inferring it in order to make this > safer: One should be able to just try to change a loop from "range" to > "prange", and either a) have things fail very hard, or b) just work > correctly and be able to trust the results. > > Note that when I suggest using NaN, it is as initial values for EACH > ITERATION, not per-thread initialization. It is not about "firstprivate" or > not, but about disabling thread-private variables entirely in favor of > "per-iteration" variables. > > I believe that by talking about "readonly" and "per-iteration" variables, > rather than "thread-shared" and "thread-private" variables, this can be used > much more safely and with virtually no knowledge of the details of > threading. Again, what's in my mind are scientific programmers with (too) > little training. > > In the end it's a matter of taste and what is most convenient to more users. > But I believe the case of needing real thread-private variables that > preserves per-thread values across iterations (and thus also can possibly > benefit from firstprivate) is seldomly enough used that an explicit > declaration is OK, in particular when it buys us so much in safety in the > common case. > > To be very precise, > > cdef double x, z > for i in prange(n): > x = f(x) > z = f(i) > ... > > goes to > > cdef double x, z > for i in prange(n): > x = z = nan > x = f(x) > z = f(i) > ... > > and we leave it to the C compiler to (trivially) optimize away "z = nan". > And, yes, it is a stopgap solution until we've got control flow analysis so > that we can outright disallow such uses of x (without threadprivate > declaration, which also gives firstprivate behaviour). OK, I had totally missed that these are per-iteration. In that case, it makes more sense. >> The "cdef threadlocal(int) foo" declaration syntax feels odd to me... >> We also probably want some way of explicitly marking a variable as >> shared and still be able to assign to/flush/sync it. Perhaps the >> parallel context could be used for these declarations, i.e. >> >> with parallel(threadlocal=a, shared=(b,c)): >> ... >> >> which would be considered an "expert" usecase. > > I'm not set on the syntax for threadlocal variables; although your proposal > feels funny/very unpythonic, almost like a C macro. For some inspiration, > here's the Python solution (with no obvious place to put the type): > > import threading > mydata = threading.local() > mydata.myvar = ... # value is threadprivate That's nice and Pythonic, though I'm not sure how we would handle typing and the passing of "mydata" around if we wanted to go that route. We have cython.locals, we could introduce cython.parallel.threadlocals(a=int), though this is a bit magical as well. >> For all the discussion of threadsavailable/threadid, the most common >> usecase I see is for allocating a large shared buffer and partitioning >> it. This seems better handled by allocating separate thread-local >> buffers, no? I still like the context idea, but everything in a >> parallel block before and after the loop(s) also seems like a natural >> place to put any setup/teardown code (though the context has the >> advantage that __exit__ is always called, even if exceptions are >> raised, which makes cleanup a lot easier to handle). > > I'd *really* like to have try/finally available in cython.parallel block for > this, although I realize that may have to wait for a while. A big part of > our discussions at the workshop were about how to handle exceptions; I guess > there'll be a "phase 2" of this where break/continue/raise is dealt with. Yeah, this is definitely an (important) second or third phase. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] speed.pypy.org
On Tue, Apr 26, 2011 at 7:50 AM, Stefan Behnel wrote: > Stefan Behnel, 15.04.2011 22:20: >> >> Stefan Behnel, 11.04.2011 15:08: >>> >>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython >>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it >>> works out well, we may have it up in a couple of days. >> >> ... or maybe not. It may take a little longer due to lack of time on his >> side. >> >> >>> I would expect that Cython won't be a big winner in this game, given that >>> it will only compile plain untyped Python code. It's also going to fail >>> entirely in some of the benchmarks. But I think it's worth having it up >>> there, simply as a way for us to see where we are performance-wise and to >>> get quick (nightly) feed-back about optimisations we try. The benchmark >>> suite is also a nice set of real-world Python code that will allow us to >>> find compliance issues. >> >> Ok, here's what I have so far. I fixed a couple of bugs in Cython and got >> at least some of the benchmarks running. Note that they are actually >> simple >> ones, only a single module. Basically all complex benchmarks fail due to >> known bugs, such as Cython def functions not accepting attribute >> assignments (e.g. on wrapping). There's also a problem with code that uses >> platform specific names conditionally, such as WindowsError when running >> on >> Windows. Cython complains about non-builtin names here. I'm considering to >> turn that into a visible warning instead of an error, so that the name >> would instead be looked up dynamically to let the code fail at runtime >> *iff* it reaches the name lookup. >> >> Anyway, here are the numbers. I got them with "auto_cpdef" enabled, >> although that doesn't even seem to make that a big difference. The >> baseline >> is a self-compiled Python 2.7.1+ (about a month old). > > [numbers stripped] > > And here's the shiny graph: > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/lastSuccessfulBuild/artifact/chart.html > > It gets automatically rebuilt by this Hudson job: > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/ Cool. Any history stored/displayed? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Tue, Apr 26, 2011 at 7:25 AM, mark florisson wrote: > On 21 April 2011 20:13, Dag Sverre Seljebotn > wrote: >> On 04/21/2011 10:37 AM, Robert Bradshaw wrote: >>> >>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson >>> wrote: >>>> >>>> On 18 April 2011 16:41, Dag Sverre Seljebotn >>>> wrote: >>>>> >>>>> Excellent! Sounds great! (as I won't have my laptop for some days I >>>>> can't >>>>> have a look yet but I will later) >>>>> >>>>> You're right about (the current) buffers and the gil. A testcase >>>>> explicitly >>>>> for them would be good. >>>>> >>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better >>>>> to >>>>> take a break from it at this point so that we can think more about that >>>>> and >>>>> not do anything rash; perhaps open up a specific thread on them and ask >>>>> for >>>>> more general input. Perhaps you want to take a break or task-switch to >>>>> something else (fused types?) until I can get around to review and merge >>>>> what you have so far? You'll know best what works for you though. If you >>>>> decide to implement explicit threadprivate variables because you've got >>>>> the >>>>> flow I certainly wom't object myself. >>>>> >>>> Ok, cool, I'll move on :) I already included a test with a prange and >>>> a numpy buffer with indexing. >>> >>> Wow, you're just plowing away at this. Very cool. >>> >>> +1 to disallowing nested prange, that seems to get really messy with >>> little benefit. >>> >>> In terms of the CEP, I'm still unconvinced that firstprivate is not >>> safe to infer, but lets leave the initial values undefined rather than >>> specifying them to be NaNs (we can do that as an implementation if you >>> want), which will give us flexibility to change later once we've had a >>> chance to play around with it. >> >> I don't see any technical issues with inferring firstprivate, the question >> is whether we want to. I suggest not inferring it in order to make this >> safer: One should be able to just try to change a loop from "range" to >> "prange", and either a) have things fail very hard, or b) just work >> correctly and be able to trust the results. >> >> Note that when I suggest using NaN, it is as initial values for EACH >> ITERATION, not per-thread initialization. It is not about "firstprivate" or >> not, but about disabling thread-private variables entirely in favor of >> "per-iteration" variables. >> >> I believe that by talking about "readonly" and "per-iteration" variables, >> rather than "thread-shared" and "thread-private" variables, this can be used >> much more safely and with virtually no knowledge of the details of >> threading. Again, what's in my mind are scientific programmers with (too) >> little training. >> >> In the end it's a matter of taste and what is most convenient to more users. >> But I believe the case of needing real thread-private variables that >> preserves per-thread values across iterations (and thus also can possibly >> benefit from firstprivate) is seldomly enough used that an explicit >> declaration is OK, in particular when it buys us so much in safety in the >> common case. >> >> To be very precise, >> >> cdef double x, z >> for i in prange(n): >> x = f(x) >> z = f(i) >> ... >> >> goes to >> >> cdef double x, z >> for i in prange(n): >> x = z = nan >> x = f(x) >> z = f(i) >> ... >> >> and we leave it to the C compiler to (trivially) optimize away "z = nan". >> And, yes, it is a stopgap solution until we've got control flow analysis so >> that we can outright disallow such uses of x (without threadprivate >> declaration, which also gives firstprivate behaviour). >> > Ah, I see, sure, that sounds sensible. I'm currently working on fused > types, so when I finish that up I'll return to that. >> >>> >>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me... >>> We also probably want some way of explicitly marking a variable as >>> shared and still be
Re: [Cython] Fused Types
On Tue, Apr 26, 2011 at 8:18 AM, mark florisson
wrote:
> On 26 April 2011 16:43, Stefan Behnel wrote:
>> mark florisson, 26.04.2011 16:23:
>>>
>>> I've been working a bit on fused types
>>> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
>>> generate code for every permutation of specific types. Currently it
>>> only works for cdef functions. Now in SimpleCallNode I'm trying to use
>>> PyrexTypes.best_match() to find the function with the best signature,
>>> but it doesn't seem to take care of coercions, e.g. it calls
>>> 'assignable_from' on the dst_type (e.g. char *), but for
>>> BuiltinObjectType (a str) this returns false.
>>
>> Which is correct. "char*" cannot coerce from/to "str". It can coerce to
>> "bytes", though.
>>
>> http://wiki.cython.org/enhancements/stringliterals
>
> Right, I see, so the thing is that I was using string literals which
> should be inferred as the bytes type when they are assigned to a char
> *. The thing is that because the type is fused, the argument may be
> anything, so I was hoping best_match would figure it out for me.
> Apparently it doesn't, and indeed, this example doesn't work:
>
> cdef extern from "Foo.h":
> cdef cppclass Foo:
> Foo(char *)
> Foo(int)
>
> cdef char *foo = "foo"
> cdef Foo* foo = new Foo("foo") # <- this doesn't work ("no suitable
> method found")
> cdef Foo* bar = new Foo(foo) # <- this works
>
> So that's pretty lame, I think I should fix that.
Agreed.
>>> Why is this the case,
>>> don't calls to overloaded C++ methods need to dispatch properly here
>>> also?
>>
>> If this doesn't work, I assume it just isn't implemented.
>>
>>
>>> Other issues are public and api declarations. So should we support
>>> that at all? We could define a macro that calls a function that does
>>> the dispatch. So e.g. for this
>>>
>>> ctypedef cython.fused_type(typeA, typeB) dtype
>>>
>>> cdef func(dtype x):
>>> ...
>>>
>>> we would get two generated functions, say, __pyx_typeA_func and
>>> __pyx_typeB_func. So we could have a macro get_func(dtype) or
>>> something that then substitutes __pyx_get_func(#dtype), where
>>> __pyx_get_func returns the pointer to the right function based on the
>>> type names. I'm not sure we should support it, right now I just put
>>> the mangled names in the header. At least the cdef functions will be
>>> sharable between Cython implementation files.
>>
>> I'm fine with explicitly forbidding this for now. It may eventually work for
>> Python object types where we can properly dispatch, but it won't easily work
>> for C types. It may work in C++, though.
>>
>
> Ok, will do.
For the moment, putting mangled names in the header should be fine. A
macro might make sense in the long term.
Somewhat orthogonal, it could makes sense to do some dispatching on
type for cpdef functions.
>>> I also noticed that for cdef functions with optional argument it gets
>>> a struct as argument, but this struct doesn't seem to end up in the
>>> header file when the function is declared public. I believe that also
>>> the typedefs for ctypedef-ed things in the .pyx file don't make it
>>> there when used to type a cdef function's arguments. Should that be
>>> fixed?
>>
>> No, I think this should also become a compiler error. These functions are
>> not meant to be called from C code. It's a Cython convenience feature. As
>> long as it's not correctly supported on both ends of the publicly exported
>> C-API, it's best to keep users from using it at all.
>
> Ok, I'll try to make Cython issue an error for these cases.
+1
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] speed.pypy.org
On Wed, Apr 27, 2011 at 12:45 AM, Stefan Behnel wrote: > Robert Bradshaw, 26.04.2011 19:52: >> >> On Tue, Apr 26, 2011 at 7:50 AM, Stefan Behnel wrote: >>> >>> Stefan Behnel, 15.04.2011 22:20: >>>> >>>> Stefan Behnel, 11.04.2011 15:08: >>>>> >>>>> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get >>>>> Cython >>>>> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it >>>>> works out well, we may have it up in a couple of days. >>>> >>>> ... or maybe not. It may take a little longer due to lack of time on his >>>> side. >>>> >>>> >>>>> I would expect that Cython won't be a big winner in this game, given >>>>> that >>>>> it will only compile plain untyped Python code. It's also going to fail >>>>> entirely in some of the benchmarks. But I think it's worth having it up >>>>> there, simply as a way for us to see where we are performance-wise and >>>>> to >>>>> get quick (nightly) feed-back about optimisations we try. The benchmark >>>>> suite is also a nice set of real-world Python code that will allow us >>>>> to >>>>> find compliance issues. >>>> >>>> Ok, here's what I have so far. I fixed a couple of bugs in Cython and >>>> got >>>> at least some of the benchmarks running. Note that they are actually >>>> simple >>>> ones, only a single module. Basically all complex benchmarks fail due to >>>> known bugs, such as Cython def functions not accepting attribute >>>> assignments (e.g. on wrapping). There's also a problem with code that >>>> uses >>>> platform specific names conditionally, such as WindowsError when running >>>> on >>>> Windows. Cython complains about non-builtin names here. I'm considering >>>> to >>>> turn that into a visible warning instead of an error, so that the name >>>> would instead be looked up dynamically to let the code fail at runtime >>>> *iff* it reaches the name lookup. >>>> >>>> Anyway, here are the numbers. I got them with "auto_cpdef" enabled, >>>> although that doesn't even seem to make that a big difference. The >>>> baseline is a self-compiled Python 2.7.1+ (about a month old). >>> >>> [numbers stripped] >>> >>> And here's the shiny graph: >>> >>> >>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/lastSuccessfulBuild/artifact/chart.html >>> >>> It gets automatically rebuilt by this Hudson job: >>> >>> >>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/ >> >> Cool. Any history stored/displayed? > > No. Also, the variances are rather large depending on the load of the > machine. Of course, that would make the history rather than a snapshot all the more useful. > Hudson/Jenkins and all its subprocesses run with a high CPU > niceness and I/O niceness, so don't expect reproducible results. > > Actually, if we want a proper history, I'd suggest a separate codespeed > installation somewhere. Makes sense. How many CPU-hours does it take? If it's not to intensive, we could probably run it, say, daily as a normal-priority job. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote: > So I fixed all that, but I'm currently wondering about the proposed > cython.typeof(). I believe it currently returns a string with the type > name, and not the type itself. Yes. This is just because there's not really anything better to return at this point. We should "fix" this at some point in the future. > So I think it would be inconsistent to > suddenly start allowing comparison with 'is' and 'isinstance' and > such. I'm open to other suggestions, but would like an expression that resolves at compile time to true/false (and we need to do branch pruning on it). Note that type() is not good enough, because it has different semantics, i.e. cdef object o = [] typeof(o), type(o) so lets not touch that one. > I'm also wondering if it would be useful to allow actual type > retrieval, which could be used in declarations and casts. For instance > consider fusing two structs with the same attribute name but different > attribute types. Perhaps in your code you want to introduce a variable > compatible with such a type, e.g. consider this: > > ctypdef struct A: > int attrib > > ctypedef struct B: > char *attrib > > ctypedef cython.fused_type(A, B) struct_t > > cdef func(struct_t mystruct, int i): > cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i > ... > > What do you think? Yes, I was thinking that would be supported as well. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] using generators
On Wed, Apr 27, 2011 at 11:42 PM, Stefan Behnel wrote: > Robert Bradshaw, 27.04.2011 22:41: >> >> On Wed, Apr 27, 2011 at 1:19 PM, Mad wrote: >>> >>> Hi list, >>> i just read, that generators are supported in cython 0.15 - great news >>> and brw, manny thanks. but is it somehow possible to use them with >>> cython 0.14 too? >>> It would be nice, if there's a patch (or the patch of 0.14 -> 0.15 >>> could be applied to the 0.14 source) to get them working, If this is >>> not possible, >> >> No, that's not possible--there's a lot that went on between here and >> there (it's not something you can just patch in). Any reason you're >> trying to stick with 0.14? >> >>> is 0.15 in a usable state? and how could i get it? >> >> It hasn't been released yet, and the development branch (on github) is >> relatively stable but not quite ready yet. As for the state of >> generators, Vitja and Stefan can comment more than I can, but there >> are still some corner cases with exception throwing that don't quite >> work like Python, and there's also a regression with respect to inline >> generators. I'm not sure if either of these is enough to be a blocker >> for a release. > > Don't think they are. As you said, the exception handling is for corner > cases (which exceptions should always be anyway) and I consider the > behaviour "close enough" now to not be a major issue. Vitja can comment on > this, too. The only case where I know that it currently diverges is > exception chaining in Python 3. And that should be fixable. > > Inline generators will require some work to become usable again. > Specifically, the expression scope that I had previously implemented for > comprehensions (and reused for generator expressions) doesn't work with the > then already created function scope of the generator. This needs to be fixed > somehow, either by moving the declarations over to the expression scope or > by directly using the function scope instead. In the latter case, it would > also be nice to merge this with the scoped comprehensions as well. > > So, agreed that these are regressions, but certainly less important than > getting generators and coroutines out in the wild. OK, I've added the inline generator tests to the bug list so we can see where we stand. I'd like to get Sage back up and compiling as well. Any thoughts on a release timeline? Any other known instabilities? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Thu, Apr 28, 2011 at 1:31 PM, mark florisson wrote: > On 28 April 2011 22:12, Robert Bradshaw wrote: >> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson >> wrote: >> >>> So I fixed all that, but I'm currently wondering about the proposed >>> cython.typeof(). I believe it currently returns a string with the type >>> name, and not the type itself. >> >> Yes. This is just because there's not really anything better to return >> at this point. We should "fix" this at some point in the future. >> >>> So I think it would be inconsistent to >>> suddenly start allowing comparison with 'is' and 'isinstance' and >>> such. >> >> I'm open to other suggestions, but would like an expression that >> resolves at compile time to true/false (and we need to do branch >> pruning on it). Note that type() is not good enough, because it has >> different semantics, i.e. >> >> cdef object o = [] >> typeof(o), type(o) >> >> so lets not touch that one. > > Right, so for fused types I don't mind string comparison with > cython.typeof(), but retrieval of the actual type for casts and > declaration remains open. I'd be fine with something like > cython.gettype(). I'd rather re-use typeof here than introduce yet another special method. On the other hand, allowing parentheses in a type declaration requires complicating the syntax even more. I'm not sure this is needed, couldn't one do cdef foo(fused_type a): cdef fused_type b = a + func() and all instances of fused_type get specialized in the body. > On the other hand, the user could already do this thing by manually > declaring another fused type that would perhaps be resolved based on > its usage. e.g. for my original example: > > ctypedef char *string_t > ctypedef cython.fused_type(int, string_t) attrib_t > > cdef func(struct_t mystruct, int i): > cdef attrib_t var = mystruct.attrib + i > > Is this something that should be supported anyway? OK, I take back what I said, I was looking at the RHS, not the LHS. If one needs to specialize in this manner, explicitly creating two branches should typically be enough. The same for casting. The one exception (perhaps) is "my_fused_type complex." Otherwise it's starting to feel too much like C++ template magic and complexity for little additional benefit. > Currently fused > types can only be used when appearing as argument types (in the > function body and as return value). >>> I'm also wondering if it would be useful to allow actual type >>> retrieval, which could be used in declarations and casts. For instance >>> consider fusing two structs with the same attribute name but different >>> attribute types. Perhaps in your code you want to introduce a variable >>> compatible with such a type, e.g. consider this: >>> >>> ctypdef struct A: >>> int attrib >>> >>> ctypedef struct B: >>> char *attrib >>> >>> ctypedef cython.fused_type(A, B) struct_t >>> >>> cdef func(struct_t mystruct, int i): >>> cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i >>> ... >>> >>> What do you think? >> >> Yes, I was thinking that would be supported as well. >> >> - Robert >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Thu, Apr 28, 2011 at 2:29 PM, mark florisson wrote: > On 28 April 2011 22:31, mark florisson wrote: >> On 28 April 2011 22:12, Robert Bradshaw wrote: >>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson >>> wrote: >>> >>>> So I fixed all that, but I'm currently wondering about the proposed >>>> cython.typeof(). I believe it currently returns a string with the type >>>> name, and not the type itself. >>> >>> Yes. This is just because there's not really anything better to return >>> at this point. We should "fix" this at some point in the future. >>> >>>> So I think it would be inconsistent to >>>> suddenly start allowing comparison with 'is' and 'isinstance' and >>>> such. >>> >>> I'm open to other suggestions, but would like an expression that >>> resolves at compile time to true/false (and we need to do branch >>> pruning on it). Note that type() is not good enough, because it has >>> different semantics, i.e. >>> >>> cdef object o = [] >>> typeof(o), type(o) >>> >>> so lets not touch that one. >> >> Right, so for fused types I don't mind string comparison with >> cython.typeof(), but retrieval of the actual type for casts and >> declaration remains open. I'd be fine with something like >> cython.gettype(). > > It seems that this isn't optimized yet, but it looks to me like it > wouldn't be very hard to do so. At least == and != could be resolved > at compile time if the other operand is a string literal. Yes. We could consider supporting "typeof(x) is typeof(double)" or even "typeof(int*)" etc. as well to not tie ourselves to strings. Or perhaps some other syntax we haven't thought of. Alternatively, it would be nice if it involved the literal fused_type as that's what we're really branching on, e.g. ctypedef cython.fused_type(A, B) AorB cdef foo(AorB x): if AorB[A]: # or .A or something ... it's hard to come up with something that plays nicely with pointer types. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel wrote: > mark florisson, 28.04.2011 23:29: >> >> On 28 April 2011 22:31, mark florisson wrote: >>> >>> On 28 April 2011 22:12, Robert Bradshaw wrote: >>>> >>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote: >>>> >>>>> So I fixed all that, but I'm currently wondering about the proposed >>>>> cython.typeof(). I believe it currently returns a string with the type >>>>> name, and not the type itself. >>>> >>>> Yes. This is just because there's not really anything better to return >>>> at this point. We should "fix" this at some point in the future. >>>> >>>>> So I think it would be inconsistent to >>>>> suddenly start allowing comparison with 'is' and 'isinstance' and >>>>> such. >>>> >>>> I'm open to other suggestions, but would like an expression that >>>> resolves at compile time to true/false (and we need to do branch >>>> pruning on it). Note that type() is not good enough, because it has >>>> different semantics, i.e. >>>> >>>> cdef object o = [] >>>> typeof(o), type(o) >>>> >>>> so lets not touch that one. >>> >>> Right, so for fused types I don't mind string comparison with >>> cython.typeof(), but retrieval of the actual type for casts and >>> declaration remains open. I'd be fine with something like >>> cython.gettype(). >> >> It seems that this isn't optimized yet, but it looks to me like it >> wouldn't be very hard to do so. At least == and != could be resolved >> at compile time if the other operand is a string literal. > > Well, the obvious place where this would happen for free would be constant > folding. But that runs *way* before type analysis, so all it sees is the > typeof() call, not the string. Actually, to support code like ctypedef cython.fused_type(object, double) my_fused_type cdef foo(my_fused_type x): if my_fused_type is object: print x.attr else: cdef my_fused_type *ptr = &x we need to resolve this branch and prune "dead code" before type analysis, so I'm thinking it may need to be a special phase, perhaps part of the fused-type-specialization phase. Here's another idea: cdef foo(numeric x): if numeric in floating: ... elif numeric is long: ... else: ... print numeric is double # after the specialization pass, this would be a literal True/False node. Again, this would all be resolved before type analysis. In terms of syntactically supporting pointers, we could either support "fused_type is typeof(void*)" or require typedefs for types not representable by an ExprNode. (This would make declaring fused types syntactically simpler as well...) I prefer the latter. - Robert > Optimising this would thus basically require a second (simpler?) constant > folding step, or at least an additional hook during (or after) type analysis > that does the comparison. I wouldn't mind too much, given that type analysis > can end up injecting some rather handy information into the tree. So a > second (even complete) constant folding step *after* type analysis may still > introduce some nice optimisations. > > We may actually consider splitting constant folding into two steps - one > that calculates the results as early as possible (and maybe does only > safe&obvious tree replacements, e.g. boolean values, strings, etc.) and a > second step that replaces subtrees by constant nodes once it knows the final > type of the result. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] What *is* a fused type?
On Fri, Apr 29, 2011 at 4:59 PM, Greg Ewing wrote: > I seem to have missed the beginning of the discussion about this > fused type business. Is there a document somewhere describing > what a "fused type" is and what it's meant to be used for? http://wiki.cython.org/enhancements/fusedtypes In short, it's a very basic type parameterization. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Fri, Apr 29, 2011 at 8:04 AM, mark florisson wrote: > On 29 April 2011 06:32, Robert Bradshaw wrote: >> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel wrote: >>> mark florisson, 28.04.2011 23:29: >>>> >>>> On 28 April 2011 22:31, mark florisson wrote: >>>>> >>>>> On 28 April 2011 22:12, Robert Bradshaw wrote: >>>>>> >>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote: >>>>>> >>>>>>> So I fixed all that, but I'm currently wondering about the proposed >>>>>>> cython.typeof(). I believe it currently returns a string with the type >>>>>>> name, and not the type itself. >>>>>> >>>>>> Yes. This is just because there's not really anything better to return >>>>>> at this point. We should "fix" this at some point in the future. >>>>>> >>>>>>> So I think it would be inconsistent to >>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and >>>>>>> such. >>>>>> >>>>>> I'm open to other suggestions, but would like an expression that >>>>>> resolves at compile time to true/false (and we need to do branch >>>>>> pruning on it). Note that type() is not good enough, because it has >>>>>> different semantics, i.e. >>>>>> >>>>>> cdef object o = [] >>>>>> typeof(o), type(o) >>>>>> >>>>>> so lets not touch that one. >>>>> >>>>> Right, so for fused types I don't mind string comparison with >>>>> cython.typeof(), but retrieval of the actual type for casts and >>>>> declaration remains open. I'd be fine with something like >>>>> cython.gettype(). >>>> >>>> It seems that this isn't optimized yet, but it looks to me like it >>>> wouldn't be very hard to do so. At least == and != could be resolved >>>> at compile time if the other operand is a string literal. >>> >>> Well, the obvious place where this would happen for free would be constant >>> folding. But that runs *way* before type analysis, so all it sees is the >>> typeof() call, not the string. >> >> Actually, to support code like >> >> ctypedef cython.fused_type(object, double) my_fused_type >> >> cdef foo(my_fused_type x): >> if my_fused_type is object: >> print x.attr >> else: >> cdef my_fused_type *ptr = &x >> >> we need to resolve this branch and prune "dead code" before type >> analysis, so I'm thinking it may need to be a special phase, perhaps >> part of the fused-type-specialization phase. Here's another idea: >> >> cdef foo(numeric x): >> if numeric in floating: >> ... >> elif numeric is long: >> ... >> else: >> ... >> print numeric is double # after the specialization pass, this >> would be a literal True/False node. >> >> Again, this would all be resolved before type analysis. In terms of >> syntactically supporting pointers, we could either support "fused_type >> is typeof(void*)" or require typedefs for types not representable by >> an ExprNode. (This would make declaring fused types syntactically >> simpler as well...) I prefer the latter. >> >> - Robert >> > > I made both things work, you can use both typeof() on expressions and > you can compare types using 'is' and 'in'. Cool. > However, with typeof() it > doesn't remove branches, so if you're doing incompatible stuff you > need to do the type matching. It could be supported, but currently it > isn't because TransformBuiltinMethods runs after > AnalyseDeclarationsTransform (for which the reason is not immediately > apparent). For one thing, it's the declaration analysis that allows you to figure out what name nodes are built-ins. > With the type matching it matches on exactly 'if src_type is > dst_type:' so you can't use 'and' and such... perhaps I should turn > these expression into a node with the constant value first and then > see if the result of the entire expression is known at compile time? Yes, that's exactly what I was thinking you should do. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Fri, Apr 29, 2011 at 3:53 AM, mark florisson wrote: > On 29 April 2011 12:28, Pauli Virtanen wrote: >> Fri, 29 Apr 2011 11:30:19 +0200, mark florisson wrote: >>> On 29 April 2011 11:03, Pauli Virtanen wrote: >> [clip] >>>> Are you planning to special-case the "real_t complex" syntax? Shooting >>>> from the sidelines, one more generic solution might be, e.g., >>> >>> I'm sorry, I'm not sure what syntax you are referring to. Are you >>> talking about actual complex numbers? >> >> This: >> >> On 28 April 2011 23:30, Robert Bradshaw >> wrote: >>> OK, I take back what I said, I was looking at the RHS, not the LHS. If >>> one needs to specialize in this manner, explicitly creating two >>> branches should typically be enough. The same for casting. The one >>> exception (perhaps) is "my_fused_type complex." Otherwise it's >>> starting to feel too much like C++ template magic and complexity for >>> little additional benefit. >> >> That is, declaring a complex type matching a real one. > > Ah, I see what you mean now. > >>>> ctypedef cython.fused_type(A, B) struct_t >>>> ctypedef cython.fused_type(float, double, paired=struct_t) real_t >>>> ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t >>>> >>>> and just restrict the specialization to cases that make sense. >>> >>> The paired means you're declaring types of attributes? >> >> No, just that real_t is specialized to float whenever struct_t is specialized >> to A and to double when B. Or a more realistic example, >> >> ctypedef cython.fused_type(float, double) real_t >> ctypedef cython.fused_type(float complex, double complex) complex_t >> >> cdef real_plus_one(complex_t a): >> real_t b = a.real >> return b + 1 >> >> which I suppose would not be a very unusual thing in numerical codes. >> This would also allow writing the case you had earlier as >> >> cdef cython.fused_type(string_t, int, paired=struct_t) attr_t >> >> cdef func(struct_t mystruct, int i): >> cdef attr_t var >> >> if typeof(mystruct) is typeof(int): >> var = mystruct.attrib + i >> ... >> else: >> var = mystruct.attrib + i >> ... >> >> Things would need to be done explicitly instead of implicitly, though, >> but it would remove the need for any special handling of >> the "complex" keyword. If we're going to introduce pairing, another option would be ctypedef fused_type((double complex, double), (float complex, float)) (complex_t, real_t) though I'm not sure I like that either. We're not trying to create the all-powerful templating system here, and anything that can be done with pairing can be done (though less elegantly) via branching on the types, or, as Pauli mentions, using a wider type is often (but not always) a viable option. We talked of supporting fused types in classes, one could do the same for structs, thus cdef fused_type(int, double) int_or_double cdef my_struct: int_or_double attr cdef func1(int_or_double x): cdef my_struct my_struct.attr = x and cdef func2(my_struct s) cdef int_or_double x = s.attr could work. Maybe we'd require an explicit "my_struct[int_or_double]". This would solve a large number of the remaining cases, and complex is like a struct + native arithmetic/promotion. > I see, so it's like a mapping. So, I didn't realize that you can't do this: > > def func(arbitrary_type complex x): > ... > > But if we just allow that for fused types, then couldn't we simply do > > ctypedef cython.fused_type(float, double) real_t > > cdef real_plus_one(real_t complex a): > real_t b = a.real > return b + 1 > > ? Then you don't need to pair anything. That's what I was thinking. It's a bit special, but on the other hand a very natural pairing that one would want to be able to do with a very natural syntax (no one reading that code would wonder what it means). > Perhaps we could introduce > real_t as a type, just like numeric and floating. So I guess > special-casing complex sounds fine with me. Perhaps real_t should be > builtin (not as an attribute of the Cython module), so the parser can > just recognize it immediately? I'd rather not add new keywords to the language, and I don't see what advantage letting real_t be known to the parser woud be, we can ship a pxd file with the common fused types. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Sat, Apr 30, 2011 at 12:51 AM, Dag Sverre Seljebotn wrote: > On 04/30/2011 08:39 AM, Robert Bradshaw wrote: >> >> On Fri, Apr 29, 2011 at 3:53 AM, mark florisson >> wrote: >>> >>> On 29 April 2011 12:28, Pauli Virtanen wrote: >>>> >>>> No, just that real_t is specialized to float whenever struct_t is >>>> specialized >>>> to A and to double when B. Or a more realistic example, >>>> >>>> ctypedef cython.fused_type(float, double) real_t >>>> ctypedef cython.fused_type(float complex, double complex) >>>> complex_t >>>> >>>> cdef real_plus_one(complex_t a): >>>> real_t b = a.real >>>> return b + 1 >>>> >>>> which I suppose would not be a very unusual thing in numerical codes. >>>> This would also allow writing the case you had earlier as >>>> >>>> cdef cython.fused_type(string_t, int, paired=struct_t) attr_t >>>> >>>> cdef func(struct_t mystruct, int i): >>>> cdef attr_t var >>>> >>>> if typeof(mystruct) is typeof(int): >>>> var = mystruct.attrib + i >>>> ... >>>> else: >>>> var = mystruct.attrib + i >>>> ... >>>> >>>> Things would need to be done explicitly instead of implicitly, though, >>>> but it would remove the need for any special handling of >>>> the "complex" keyword. >> >> If we're going to introduce pairing, another option would be >> >> ctypedef fused_type((double complex, double), (float complex, >> float)) (complex_t, real_t) >> >> though I'm not sure I like that either. We're not trying to create the >> all-powerful templating system here, and anything that can be done >> with pairing can be done (though less elegantly) via branching on the >> types, or, as Pauli mentions, using a wider type is often (but not >> always) a viable option. > > Keeping the right balance is difficult. But, at least there's some cases of > needing this in various codebases when interfacing with LAPACK. > > Most uses of templating with Cython code I've seen so far does a similar > kind of "zip" as what you have above (as we discussed on the workshop). So > at least the usage pattern you write above is very common. > > float32 is not about to disappear, it really is twice as fast when you're > memory IO bound. > > Using a wider type is actually quite often not possible; any time the type > is involved as the base type of an array it is not possible, and that's a > pretty common case. (With LAPACK you take the address of the variable and > pass it to Fortran, so using a wider type is not possible there either, > although I'll agree that's a more remote case.) > > My proposal: Don't support either "real_t complex" or paired fused types for > the time being. Then see. +1, then we can address real need. > But my vote is for paired fused types instead of "real_t complex". > > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Sat, Apr 30, 2011 at 7:24 AM, Stefan Behnel wrote: > Robert Bradshaw, 29.04.2011 06:32: >> >> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel >> wrote: >>> >>> mark florisson, 28.04.2011 23:29: >>>> >>>> On 28 April 2011 22:31, mark florisson wrote: >>>>> >>>>> On 28 April 2011 22:12, Robert Bradshaw wrote: >>>>>> >>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote: >>>>>> >>>>>>> So I fixed all that, but I'm currently wondering about the proposed >>>>>>> cython.typeof(). I believe it currently returns a string with the >>>>>>> type >>>>>>> name, and not the type itself. >>>>>> >>>>>> Yes. This is just because there's not really anything better to return >>>>>> at this point. We should "fix" this at some point in the future. >>>>>> >>>>>>> So I think it would be inconsistent to >>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and >>>>>>> such. >>>>>> >>>>>> I'm open to other suggestions, but would like an expression that >>>>>> resolves at compile time to true/false (and we need to do branch >>>>>> pruning on it). Note that type() is not good enough, because it has >>>>>> different semantics, i.e. >>>>>> >>>>>> cdef object o = [] >>>>>> typeof(o), type(o) >>>>>> >>>>>> so lets not touch that one. >>>>> >>>>> Right, so for fused types I don't mind string comparison with >>>>> cython.typeof(), but retrieval of the actual type for casts and >>>>> declaration remains open. I'd be fine with something like >>>>> cython.gettype(). >>>> >>>> It seems that this isn't optimized yet, but it looks to me like it >>>> wouldn't be very hard to do so. At least == and != could be resolved >>>> at compile time if the other operand is a string literal. >>> >>> Well, the obvious place where this would happen for free would be >>> constant >>> folding. But that runs *way* before type analysis, so all it sees is the >>> typeof() call, not the string. >> >> Actually, to support code like >> >> ctypedef cython.fused_type(object, double) my_fused_type >> >> cdef foo(my_fused_type x): >> if my_fused_type is object: >> print x.attr >> else: >> cdef my_fused_type *ptr =&x >> >> we need to resolve this branch and prune "dead code" before type >> analysis, so I'm thinking it may need to be a special phase, perhaps >> part of the fused-type-specialization phase. Here's another idea: >> >> cdef foo(numeric x): >> if numeric in floating: >> ... >> elif numeric is long: >> ... >> else: >> ... >> print numeric is double # after the specialization pass, this >> would be a literal True/False node. >> >> Again, this would all be resolved before type analysis. > > Hmm, I wonder, if we ever want to start with whole program analysis, and we > find code like this: > > def func(arg): > ... > > a = [1,2,3] > func(a) > > we could infer that "arg" is actually a "fused_type(object, list)". So, I > think, what we eventually want is an intermingling of type analysis and > specialisation. Meaning, we'd find a new type during type analysis or > inference, split off a new version of the function and then analyse it. But > maybe that's just a pie in the sky for now. This fits into the idea of creating two (or possibly more, but avoiding combinatorial explosion) branches, a fast one and a safe one, and bailing from one to the other if guards fail (all within the same function). ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Pull request emails
Excellent, thanks. On Sat, Apr 30, 2011 at 2:35 PM, Dag Sverre Seljebotn wrote: > Finally think I figured out how to get pull request emails (thanks to Gael > V). From https://github.com/organizations/cython/teams/24445: > > """ > Owners do not receive notifications for the organization's repos by default. > To receive notifications, create a team and add the owners and repos for > which notifications are desired. > """ > > I created Reviewers and added me, Stefan & Robert for now. > > https://github.com/organizations/cython/teams/54516 > > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Sun, May 1, 2011 at 2:38 AM, mark florisson
wrote:
> On 30 April 2011 09:51, Dag Sverre Seljebotn
> wrote:
>> On 04/30/2011 08:39 AM, Robert Bradshaw wrote:
>>>
>>> On Fri, Apr 29, 2011 at 3:53 AM, mark florisson
>>> wrote:
>>>>
>>>> On 29 April 2011 12:28, Pauli Virtanen wrote:
>>>>>
>>>>> No, just that real_t is specialized to float whenever struct_t is
>>>>> specialized
>>>>> to A and to double when B. Or a more realistic example,
>>>>>
>>>>> ctypedef cython.fused_type(float, double) real_t
>>>>> ctypedef cython.fused_type(float complex, double complex)
>>>>> complex_t
>>>>>
>>>>> cdef real_plus_one(complex_t a):
>>>>> real_t b = a.real
>>>>> return b + 1
>>>>>
>>>>> which I suppose would not be a very unusual thing in numerical codes.
>>>>> This would also allow writing the case you had earlier as
>>>>>
>>>>> cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>>>>>
>>>>> cdef func(struct_t mystruct, int i):
>>>>> cdef attr_t var
>>>>>
>>>>> if typeof(mystruct) is typeof(int):
>>>>> var = mystruct.attrib + i
>>>>> ...
>>>>> else:
>>>>> var = mystruct.attrib + i
>>>>> ...
>>>>>
>>>>> Things would need to be done explicitly instead of implicitly, though,
>>>>> but it would remove the need for any special handling of
>>>>> the "complex" keyword.
>>>
>>> If we're going to introduce pairing, another option would be
>>>
>>> ctypedef fused_type((double complex, double), (float complex,
>>> float)) (complex_t, real_t)
>>>
>>> though I'm not sure I like that either. We're not trying to create the
>>> all-powerful templating system here, and anything that can be done
>>> with pairing can be done (though less elegantly) via branching on the
>>> types, or, as Pauli mentions, using a wider type is often (but not
>>> always) a viable option.
>>
>> Keeping the right balance is difficult. But, at least there's some cases of
>> needing this in various codebases when interfacing with LAPACK.
>>
>> Most uses of templating with Cython code I've seen so far does a similar
>> kind of "zip" as what you have above (as we discussed on the workshop). So
>> at least the usage pattern you write above is very common.
>>
>> float32 is not about to disappear, it really is twice as fast when you're
>> memory IO bound.
>>
>> Using a wider type is actually quite often not possible; any time the type
>> is involved as the base type of an array it is not possible, and that's a
>> pretty common case.
>
> Well, if the array is passed into the function directly (and not e.g.
> as an attribute of something passed in), then you can just write
> 'my_fused_type *array' or 'my_fused_type array[]', and the base type
> will be available as 'my_fused_type'.
>
>> (With LAPACK you take the address of the variable and
>> pass it to Fortran, so using a wider type is not possible there either,
>> although I'll agree that's a more remote case.)
>>
>> My proposal: Don't support either "real_t complex" or paired fused types for
>> the time being. Then see.
>
> Ok, sounds good.
>
>> But my vote is for paired fused types instead of "real_t complex".
>>
>> Dag Sverre
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> A remaining issue which I'm not quite certain about is the
> specialization through subscripts, e.g. func[double]. How should this
> work from Python space (assuming cpdef functions)? Would we want to
> pass in cython.double etc? Because it would only work for builtin
> types, so what about types that aren't exposed to Python but can still
> be coerced to and from Python? Perhaps it would be better to pass in
> strings instead. I also think e.g. "int *" reads better than
> cython.pointer(cython.int).
That's whey we offer cython.p_int. On that note, we should support
cython.astype("int *") or something like that. Generally, I don't like
encoding semantic information in strings.
OTHO, since it'll be a mapping of some sort, there's no reason we
can't support both. Most of the time it should dispatch (at runtime or
compile time) based on the type of the arguments.
> It also sounds bad to rely on objects from the Shadow module, as
> people using Cython modules may not have Cython (or the Shadow module
> shipped as cython) installed. And what happens if they import it under
> a different name and we import another cython module?
The (vague) idea is that one could ship as much of the cython shadow
module with your own code as needed to run without Cython.
> Perhaps the
> specializations of the signature could also be exposed on the function
> object, so users can see which ones are available.
Sure.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Mon, May 2, 2011 at 1:56 PM, mark florisson
wrote:
> On 2 May 2011 18:24, Robert Bradshaw wrote:
>> On Sun, May 1, 2011 at 2:38 AM, mark florisson
>> wrote:
>>> A remaining issue which I'm not quite certain about is the
>>> specialization through subscripts, e.g. func[double]. How should this
>>> work from Python space (assuming cpdef functions)? Would we want to
>>> pass in cython.double etc? Because it would only work for builtin
>>> types, so what about types that aren't exposed to Python but can still
>>> be coerced to and from Python? Perhaps it would be better to pass in
>>> strings instead. I also think e.g. "int *" reads better than
>>> cython.pointer(cython.int).
>>
>> That's whey we offer cython.p_int. On that note, we should support
>> cython.astype("int *") or something like that. Generally, I don't like
>> encoding semantic information in strings.
>>
>> OTHO, since it'll be a mapping of some sort, there's no reason we
>> can't support both. Most of the time it should dispatch (at runtime or
>> compile time) based on the type of the arguments.
>
> If we have an argument type that is composed of a fused type, would be
> want the indexing to specify the composed type or the fused type? e.g.
>
> ctypedef floating *floating_p
How should we support this? It's clear in this case, but only because
you chose good names. Another option would be to require
parameterization floating_p, with floating_p[floating] the
"as-yet-unparameterized" version. Explicit but redundant. (The same
applies to struct as classes as well as typedefs.) On the other had,
the above is very succinct and clear in context, so I'm leaning
towards it. Thoughts?
> cdef func(floating_p x):
> ...
>
> Then do we want
>
> func[double](10.0)
>
> or
>
> func[double_p](10.0)
>
> to specialize func?
The latter.
> FYI, the type checking works like 'double_p is
> floating_p' and not 'double is floating_p'. But for functions this is
> a little different. On the one hand specifying the full types
> (double_p) makes sense as you're kind of specifying a signature, but
> on the other hand you're specializing fused types and you don't care
> how they are composed -- especially if they occur multiple times with
> different composition. So I'm thinking we want 'func[double]'.
That's what I'm thinking too. The type you're branching on is
floating, and withing that block you can declare variables as
floating*, ndarray[dtype=floating], etc.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Fwd: automatic character conversion problem
Dear Cython developers, Recently I encountered a problem with Cython's automatic char* to string conversion (Cython version 0.14.1). I'll attach two sample source files. The first one, char2str_a.pyx prints "The quick...", just as I expected. But the second example prints "... lazy dog.". In the original situation I had a call to free() instead of the call to strcpy() which I use here for illustration purposes. Then I got unpredictable results. Apparently the Python string object keeps referring to the C char* a bit longer than I would expect. A previous version (0.11.2) didn't have this problem. Best regards, Hans Terlouw -- J. P. Terlouw Kapteyn Astronomical Institute University of Groningen Postbus 800 NL-9700 AV Groningen The Netherlands Phone: +31-(0)50-3634068/73 Fax: +31-(0)50-3636100 Web: http://www.astro.rug.nl/~terlouw/ cdef extern from "stdlib.h": void free(void* ptr) void* malloc(size_t size) cdef extern from "string.h": char *strcpy(char *dest, char *src) def char2str(): cdef char *c_str_a = malloc(80) cdef char *c_str_b = "The quick... " cdef char *c_str_c = "... lazy dog. " strcpy(c_str_a, c_str_b) p_str = c_str_a strcpy(c_str_a, c_str_c) p_str = p_str.rstrip() print p_str cdef extern from "stdlib.h": void free(void* ptr) void* malloc(size_t size) cdef extern from "string.h": char *strcpy(char *dest, char *src) def char2str(): cdef char *c_str_a = malloc(80) cdef char *c_str_b = "The quick... " cdef char *c_str_c = "... lazy dog. " strcpy(c_str_a, c_str_b) p_str = c_str_a strcpy(c_str_a, c_str_c) print p_str.rstrip() cdef extern from "stdlib.h": void free(void* ptr) void* malloc(size_t size) cdef extern from "string.h": char *strcpy(char *dest, char *src) def char2str(): cdef char *c_str_a = malloc(80) cdef char *c_str_b = "The quick... " cdef char *c_str_c = "... lazy dog. " strcpy(c_str_a, c_str_b) p_str = c_str_a strcpy(c_str_a, c_str_c) p_str = p_str.rstrip() print p_str cdef extern from "stdlib.h": void free(void* ptr) void* malloc(size_t size) cdef extern from "string.h": char *strcpy(char *dest, char *src) def char2str(): cdef char *c_str_a = malloc(80) cdef char *c_str_b = "The quick... " cdef char *c_str_c = "... lazy dog. " strcpy(c_str_a, c_str_b) p_str = c_str_a strcpy(c_str_a, c_str_c) print p_str.rstrip() ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] automatic character conversion problem
On Fri, Apr 29, 2011 at 6:57 AM, Hans Terlouw wrote: > Dear Cython developers, > > Recently I encountered a problem with Cython's automatic char* to string > conversion (Cython version 0.14.1). I'll attach two sample source files. The > first one, char2str_a.pyx prints "The quick...", just as I expected. But the > second example prints "... lazy dog.". In the original situation I had a > call to > free() instead of the call to strcpy() which I use here for illustration > purposes. Then I got unpredictable results. Apparently the Python string > object > keeps referring to the C char* a bit longer than I would expect. A previous > version (0.11.2) didn't have this problem. This is due to type inference, in the second example, p_str is inferred to be of type char*. - Robert > cdef extern from "stdlib.h": > void free(void* ptr) > void* malloc(size_t size) > > cdef extern from "string.h": > char *strcpy(char *dest, char *src) > > def char2str(): > cdef char *c_str_a = malloc(80) > cdef char *c_str_b = "The quick... " > cdef char *c_str_c = "... lazy dog. " > > strcpy(c_str_a, c_str_b) > > p_str = c_str_a > strcpy(c_str_a, c_str_c) > p_str = p_str.rstrip() > print p_str > > > > cdef extern from "stdlib.h": > void free(void* ptr) > void* malloc(size_t size) > > cdef extern from "string.h": > char *strcpy(char *dest, char *src) > > def char2str(): > cdef char *c_str_a = malloc(80) > cdef char *c_str_b = "The quick... " > cdef char *c_str_c = "... lazy dog. " > > strcpy(c_str_a, c_str_b) > > p_str = c_str_a > strcpy(c_str_a, c_str_c) > print p_str.rstrip() > > > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Tue, May 3, 2011 at 12:59 AM, mark florisson
wrote:
> On 3 May 2011 00:21, Robert Bradshaw wrote:
>> On Mon, May 2, 2011 at 1:56 PM, mark florisson
>> wrote:
>>> On 2 May 2011 18:24, Robert Bradshaw wrote:
>>>> On Sun, May 1, 2011 at 2:38 AM, mark florisson
>>>> wrote:
>>>>> A remaining issue which I'm not quite certain about is the
>>>>> specialization through subscripts, e.g. func[double]. How should this
>>>>> work from Python space (assuming cpdef functions)? Would we want to
>>>>> pass in cython.double etc? Because it would only work for builtin
>>>>> types, so what about types that aren't exposed to Python but can still
>>>>> be coerced to and from Python? Perhaps it would be better to pass in
>>>>> strings instead. I also think e.g. "int *" reads better than
>>>>> cython.pointer(cython.int).
>>>>
>>>> That's whey we offer cython.p_int. On that note, we should support
>>>> cython.astype("int *") or something like that. Generally, I don't like
>>>> encoding semantic information in strings.
>>>>
>>>> OTHO, since it'll be a mapping of some sort, there's no reason we
>>>> can't support both. Most of the time it should dispatch (at runtime or
>>>> compile time) based on the type of the arguments.
>>>
>>> If we have an argument type that is composed of a fused type, would be
>>> want the indexing to specify the composed type or the fused type? e.g.
>>>
>>> ctypedef floating *floating_p
>>
>> How should we support this? It's clear in this case, but only because
>> you chose good names. Another option would be to require
>> parameterization floating_p, with floating_p[floating] the
>> "as-yet-unparameterized" version. Explicit but redundant. (The same
>> applies to struct as classes as well as typedefs.) On the other had,
>> the above is very succinct and clear in context, so I'm leaning
>> towards it. Thoughts?
>
> Well, it is already supported. floating is fused, so any composition
> of floating is also fused.
>
>>> cdef func(floating_p x):
>>> ...
>>>
>>> Then do we want
>>>
>>> func[double](10.0)
>>>
>>> or
>>>
>>> func[double_p](10.0)
>>>
>>> to specialize func?
>>
>> The latter.
>
> I'm really leaning towards the former.
Ugh. I totally changed the meaning of that when I refactored my email.
I'm in agreement with you: func[double].
> What if you write
>
> cdef func(floating_p x, floating_p *y):
> ...
>
> Then specializing floating_p using double_p sounds slightly
> nonsensical, as you're also specializing floating_p *.
>
>>> FYI, the type checking works like 'double_p is
>>> floating_p' and not 'double is floating_p'. But for functions this is
>>> a little different. On the one hand specifying the full types
>>> (double_p) makes sense as you're kind of specifying a signature, but
>>> on the other hand you're specializing fused types and you don't care
>>> how they are composed -- especially if they occur multiple times with
>>> different composition. So I'm thinking we want 'func[double]'.
>>
>> That's what I'm thinking too. The type you're branching on is
>> floating, and withing that block you can declare variables as
>> floating*, ndarray[dtype=floating], etc.
>
> What I actually meant there was "I think we want func[double] for the
> func(floating_p x) signature".
>
> Right, people can already say 'cdef func(floating *p): ...' and then
> use 'floating'. However, if you do 'cdef floating_p x): ...', then
> 'floating' is not available, only 'floating_p'. It would be rather
> trivial to also support 'floating' in the latter case, which I think
> we should,
floating is implicitly available, we could require making it explicit.
> unless you are adamant about prohibiting regular typedefs
> of fused types.
No, I'm nto adamant against it, just wanted to get some discussion going.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Tue, May 3, 2011 at 7:06 AM, Dag Sverre Seljebotn wrote: > On 05/03/2011 03:51 PM, Stefan Behnel wrote: >> >> mark florisson, 03.05.2011 15:17: >>> >>> if you have >>> >>> cdef func(floating x, floating y): >>> ... >>> >>> you get a "float, float" version, and a "double, double" version, but >>> not "float, double" or "double, float". >> >> So, what would you have to do in order to get a "float, double" and >> "double, float" version then? Could you get that with >> >> ctypedef fused_type(double, float) floating_df >> ctypedef fused_type(float, double) floating_fd >> >> cdef func(floating_df x, floating_fd y): >> >> ? > > Well, if you do something like > > ctypedef fused_type(float, double) speed_t > ctypedef fused_type(float, double) acceleration_t > > cdef func(speed_t x, acceleration_t y) > > then you get 4 specializations. Each new typedef gives a new polymorphic > type. Yep. > OTOH, with > > ctypedef speed_t acceleration_t > > I guess only 2 specializations. > > Treating the typedefs in this way is slightly fishy of course. It may hint > that "ctypedef" is the wrong way to declare a fused type *shrug*. True, but if we start supporting nested things, I think this still makes the most sense. E.g. ctypedef floating speed_t ctypedef floating acceleration_t struct delta: speed_t v acceleration_t a would still be exactly two versions (or three, if we throw long-double in there), rather than having un-intended combinatorial explosion. The as-yet-unspecialized version of delta would be delta[floating] or, possibly, just delta. One could explicitly ask for delta[double] or delta[float]. In terms of floating_p, note that "floating is double" and "floating_p is double*" could both make perfect sense for that particular specialization. In terms of compiler support, as long as ctypedef double my_double produced something "in floating" then I think it could all be done in a header file. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Tue, May 3, 2011 at 10:06 AM, mark florisson wrote: > On 3 May 2011 18:00, Robert Bradshaw wrote: >> floating is implicitly available, we could require making it explicit. > > How would we make it explicit. Require the parameterization, i.e. floating_p[floating] would be the as-yet-unspecified type. In this particular example, it does seem unnecessarily verbose. Without it, one would have to know that cdef object foo(Vector v): ... may depend on floating if Vector does. On Tue, May 3, 2011 at 10:52 AM, Dag Sverre Seljebotn wrote: > I was wrong. We need > > cdef f(floating x, floating_p y) > > ...to get 2 specializations, not 4. And the rest follows from there. So I'm > with Robert's real stance :-) > > I don't think we want flexibility, we want simplicity over all. You can > always use a templating language. +1 > Btw we shouldn't count on pruning for the design of this, I think this will > for a large part be used with def functions. And if you use a cdef function > from another module through a pxd, you also need all versions. Well, we'll want to avoid compiler warnings. E.g. floating might include long double, but only float and double may be used. In pxd and def functions, however, we will make all versions available. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Tue, May 3, 2011 at 12:59 PM, Dag Sverre Seljebotn wrote: > On 05/03/2011 08:19 PM, Robert Bradshaw wrote: > >>> Btw we shouldn't count on pruning for the design of this, I think this >>> will >>> for a large part be used with def functions. And if you use a cdef >>> function >>> from another module through a pxd, you also need all versions. >> >> Well, we'll want to avoid compiler warnings. E.g. floating might >> include long double, but only float and double may be used. In pxd and >> def functions, however, we will make all versions available. > > Which is a reminder to hash out exactly how the dispatch will be resolved > when coming from Python space (we do want to support "f(x, y)", without > []-qualifier, when calling from Python, right?) > > Fused types mostly make sense when used through PEP 3118 memory views (using > the planned syntax for brevity): > > def f(floating[:] x, floating y): ... > > I'm thinking that in this kind of situation we let the array override how y > is interpreted (y will always be a double here, but if x is passed as a > float32 then use float32 for y as well and coerce y). > > Does this make sense as a general rule -- if there's a conflict between > array arguments and scalar arguments (array base type is narrower than the > scalar type), the array argument wins? It makes sense because we can easily > convert a scalar while we can't convert an array; and there's no "3.4f" > notation in Python. I hadn't thought of that... it''s a bit odd, but we do have object -> float but no float[:] -> double[:], so the notion of something being the only match, even if it involves truncation, would be OK here. I'd like the rules to be clear. Perhaps we allow object -> float (e.g. in the auto-dispatching mode) but not double -> float. > This makes less sense > > def f(floating x): ... > > as it can only ever resolve to double; although I guess we should allow it > for consistency with usecases that do make sense, such as "real_or_complex" > and "int_or_float" > > The final and most difficult problem is what Python ints resolve to in this > context. The widest integer type available in the fused type? Always > Py_ssize_t? -1 on making the dispatch depend on the actual run-time value. -1 to run-time cutoffs. I'd choose the widest integral type. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Tue, May 3, 2011 at 4:40 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> ctypedef fused_type(float, double) speed_t >> ctypedef fused_type(float, double) acceleration_t >> >> then you get 4 specializations. >> >> ctypedef speed_t acceleration_t >> >> I guess only 2 specializations. >> >> Treating the typedefs in this way is slightly fishy of course. > > Indeed. This whole business seems rather too implicit to > me. I think I'd rather have explicit type parameters in > some form. Maybe > > cdef func2[floating F](F x, F y): > # 2 specialisations > > cdef func4[floating F, floating G](F x, G y): > # 4 specialisations > > This also makes it clear how to refer to particular > specialisations: func2[float] or func4[float, double]. > > Pointers are handled in a natural way: > > cdef funcfp[floating F](F x, F *y): > # 2 specialisations > > It also extends naturally to fused types used in other > contexts: > > cdef struct Vector[floating F]: > F x, y, z That's an idea, it is nice and explicit without being too verbose. Any thoughts on how one would define one's own "floating?" I presume one then use a Vector[F] inside of func2[floating F]? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Wed, May 4, 2011 at 1:47 AM, mark florisson wrote: > On 4 May 2011 10:24, Dag Sverre Seljebotn wrote: >> On 05/04/2011 01:07 AM, Greg Ewing wrote: >>> >>> mark florisson wrote: >>> cdef func(floating x, floating y): ... you get a "float, float" version, and a "double, double" version, but not "float, double" or "double, float". >>> >>> It's hard to draw conclusions from this example because >>> it's degenerate. You don't really need multiple versions of a >>> function like that, because of float <-> double coercions. >>> >>> A more telling example might be >>> >>> cdef double dot_product(floating *u, floating *v, int length) >>> >>> By your current rules, this would give you one version that >>> takes two float vectors, and another that takes two double >>> vectors. >>> >>> But if you want to find the dot product of a float vector and >>> a double vector, you're out of luck. >> >> First, I'm open for your proposed syntax too...But in the interest of seeing >> how we got here: >> >> The argument to the above goes that you *should* be out of luck. For >> instance, talking about dot products, BLAS itself has float-float and >> double-double, but not float-double AFAIK. >> >> What you are saying that this does not have the full power of C++ templates. >> And the answer is that yes, this does not have the full power of C++ >> templates. >> >> At the same time we discussed this, we also discussed better support for >> string-based templating languages (so that, e.g., compilation error messages >> could refer to the template file). The two are complementary. >> >> Going back to Greg's syntax: What I don't like is that it makes the simple >> unambiguous cases, where this would actually be used in real life, less >> readable. >> >> Would it be too complicated to have both? For instance; >> >> i) You are allowed to use a *single* fused_type on a *function* without >> declaration. >> >> def f(floating x, floating *y): # ok >> >> Turns into >> >> def f[floating T](T x, T *y): >> >> This is NOT ok: >> >> def f(floating x, integral y): >> # ERROR: Please explicitly declare fused types inside [] >> >> ii) Using more than one fused type, or using it on a cdef class or struct, >> you need to use the [] declaration. >> > > I don't think it would be too complicated, but as you mention it's > probably not a very likely case, and if the user does need it, a new > (equivalent) fused type can be created. The current way reads a lot > nicer than the indexed one in my opinion. So I'd be fine with > implementing it, but I find the current way more elegant. I was actually thinking of exactly the same thing--supporting syntax (i) for the case of a single type parameter, but the drawback is the introduction of two distinct syntaxes for essentially the same feature. Something like this is necessary to give an ordering to the types for structs and classes, or when a fused type is used for intermediate results but not in the argument list. I really like the elegance of the (much more common) single-parameter variant. Another option is using the with syntax, which was also considered for supporting C++ templates. >> Finally: It is a bit uncomfortable that we seem to be hashing things out >> even as Mark is implementing this. Would it be feasible to have a Skype >> session sometimes this week where everybody interested in the outcome of >> this come together for an hour and actually decide on something? >> >> Mark: How much does this discussion of syntax impact your development? Are >> you able to treat them just as polish on top and work on the "engine" >> undisturbed by this? > > Thanks for your consideration, I admit it it feels a bit uncomfortable > :) But at least this change shouldn't have such a big impact on the > code, it would mean some changes in a select few places, so it's > definitely polish. In any event, before we settle on this, I'd like to > do the cpdef support first and work on indexing from Python space, so > I think we have enough time to settle this argument on the ML. > Before that, I'm just going to finish up for a pull request for the > OpenMP branch, I'd like to see if I can get rid of some warnings. Yes, please feel free to focus on the back end and move onto other the things while the syntax is still in limbo, rather than implementing every whim of the mailing list :). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Git workflow, branches, pull requests
On Thu, May 5, 2011 at 1:22 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 05.05.2011 21:52: >> >> There was just a messup in git history: Mark's OpenMP pull request got >> merged twice; all commits show up two times. > > What (I think) happened, was that Vitja pulled in Mark's changes into his > unreachable code removal branch, and they ended up in his pull request. I > guess I was assuming that git wouldn't care too much about branch > duplication, so I just accepted the pull request via the web interface. > Apparently, it did care. > > I tend to rebase my local change sets before pushing them, and I think it > makes sense to continue doing that. +1, I think for as-yet-unpublished changes, it makes the most sense to rebase, but for a longer-term branch, merging isn't as disruptive to the history (in fact is probably more reflective of what's going on) and is much better than duplication. To clarify, is this only a problem when we have A cloned from master B cloned from A (or from master and then pulls in A) A rebases A+B merged into master ? If this is the case, then we could simply make the rule that you should ask before hacking a clone atop anything but master. (Multiple people can share a repeatedly-rebased branch, right.) We could also us the underscore (or another) convention to mean "this branch is being used as a queue, puller beware." Surely other projects have dealt with this. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Git workflow, branches, pull requests
I don't like the default to be "don't pull from me"--I'd rather there be some convention to indicate a branch is being used as a queue. Maybe even foo-queue, or a leading underscore if people like that. On Thu, May 5, 2011 at 2:03 PM, Dag Sverre Seljebotn wrote: > Yes, that is the only time it happens. > > Do we agree on a) ask before you pull anything that is not in cython/* (ie > in private repos), b) document it in hackerguide? > > DS > > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > > Robert Bradshaw wrote: >> >> On Thu, May 5, 2011 at 1:22 PM, Stefan Behnel wrote: >> > Dag Sverre Seljebotn, 05.05.2011 21:52: >> >> There was just a messup in >> git history: Mark's OpenMP pull request got >> merged twice; all commits >> show up two times. > > What (I think) happened, was that Vitja pulled in >> Mark's changes into his > unreachable code removal branch, and they ended up >> in his pull request. I > guess I was assuming that git wouldn't care too >> much about branch > duplication, so I just accepted the pull request via the >> web interface. > Apparently, it did care. > > I tend to rebase my local >> change sets before pushing them, and I think it > makes sense to continue >> doing that. +1, I think for as-yet-unpublished changes, it makes the most >> sense to rebase, but for a longer-term branch, merging isn't as disruptive >> to the history (in fact is probably more reflective of what's going on) and >> is much better than duplication. To clarify, is this only a problem when we >> have A cloned from master B cloned from A (or from master and then pulls in >> A) A rebases A+B merged into master ? If this is the case, then we could >> simply make the rule that you should ask before hacking a clone atop >> anything but master. (Multiple people can share a repeatedly-rebased branch, >> right.) We could also us the underscore (or another) convention to mean >> "this branch is being used as a queue, puller beware." Surely other projects >> have dealt with this. - Robert >> >> cython-devel mailing list cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
